{"type":"ScholarlyArticle","title":"Learning Multi-Objective Rewards and User Utility Function in Contextual Bandits for Personalized Ranking","authors":[{"name":"Nirandika Wanigasekara","slug":"nirandika-wanigasekara","given":"Nirandika","family":"Wanigasekara","affiliation":"National University of Singapore"},{"name":"Yuxuan Liang","slug":"yuxuan-liang","given":"Yuxuan","family":"Liang","affiliation":"National University of Singapore"},{"name":"Siong Thye Goh","slug":"siong-thye-goh","given":"Siong Thye","family":"Goh","affiliation":"Singapore Management University"},{"name":"Ye Liu","slug":"ye-liu","given":"Ye","family":"Liu","affiliation":"National University of Singapore"},{"name":"Joseph Jay Williams","slug":"joseph-jay-williams","given":"Joseph Jay","family":"Williams","affiliation":"University of Toronto"},{"name":"David S. Rosenblum","slug":"david-s-rosenblum","given":"David S.","family":"Rosenblum","affiliation":"National University of Singapore"}],"abstract":"This paper tackles the problem of providing users with ranked lists of relevant search results, by incorporating contextual features of the users and search results, and learning how a user values multiple objectives. For example, to recommend a ranked list of hotels, an algorithm must learn which hotels are the right price for users, as well as how users vary in their weighting of price against the location. \r\nIn our paper, we formulate the context-aware, multi-objective, ranking problem as a Multi-Objective Contextual Ranked Bandit (MOCR-B). \r\nTo solve the MOCR-B problem, we present a novel algorithm, named Multi-Objective Utility-Upper Confidence Bound (MOU-UCB).\r\nThe goal of MOU-UCB is to learn how to generate a ranked list of resources that maximizes the rewards in multiple objectives to give relevant search results. Our algorithm learns to predict rewards in multiple objectives based on contextual information (combining the Upper Confidence Bound algorithm for multi-armed contextual bandits with neural network embeddings), as well as learns how a user weights the multiple objectives. Our empirical results reveal that the ranked lists generated by MOU-UCB lead to better click-through rates, compared to approaches that do not learn the utility function over multiple reward objectives.","doi":"10.24963/ijcai.2019/532","url":"https://ijcai.org/proceedings/2019/532","pdf":"https://ijcai.org/proceedings/2019/0532.pdf","markdown":"https://ijcai.org/proceedings/2019/532.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/532","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning M-Z","track":"Main track","number":532,"pages":{"first":3835,"last":3841},"keywords":["Machine Learning Applications: Applications of Reinforcement Learning","Machine Learning: Online Learning"],"videos":[],"references":[{"position":1,"raw":"[Auer et al., 2016] Peter Auer, Chao-Kai Chiang, Ronald Ortner, and Madalina Drugan. Pareto front identification from stochastic bandit feedback. In Artificial Intelligence and Statistics, pages 939-947, 2016.","title":"Pareto front identification from stochastic bandit feedback","authors":"Auer","year":"2016","venue":"Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Drugan and Nowé, 2013] Madalina M Drugan and Ann Nowé. Designing multi-objective multi-armed bandits al- gorithms: A study. In Neural Networks (IJCNN), The 2013 International Joint Conference on, pages 1-8. IEEE, 2013.","title":"Designing multi-objective multi-armed bandits al-gorithms: A study","authors":"M Madalina, Ann Drugan, Nowé","year":"2013","venue":"Neural Networks (IJCNN), The 2013 International Joint Conference on","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Drugan and Nowé, 2014] Madalina M Drugan and Ann Nowé. Scalarization based pareto optimal set of arms identification algorithms. In Neural Networks (IJCNN), 2014 International Joint Conference on, pages 2690- 2697. IEEE, 2014.","title":"Scalarization based pareto optimal set of arms identification algorithms","authors":"M Madalina, Ann Drugan, Nowé","year":"2014","venue":"Neural Networks (IJCNN), 2014 International Joint Conference on","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Gal and Ghahramani, 2016] Yarin Gal and Zoubin Ghahra- mani. A theoretically grounded application of dropout in recurrent neural networks. In Advances in neural informa- tion processing systems, pages 1019-1027, 2016.","title":"Yarin Gal and Zoubin Ghahramani. A theoretically grounded application of dropout in recurrent neural networks","authors":"Ghahramani Gal","year":"2016","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Kingma and Ba, 2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Lacerda, 2015] Anisio Lacerda. Contextual bandits for multi-objective recommender systems. In Intelligent Sys- tems (BRACIS), 2015 Brazilian Conference on, pages 68- 73. IEEE, 2015.","title":"Anisio Lacerda. Contextual bandits for multi-objective recommender systems","authors":"Lacerda","year":"2015","venue":"Intelligent Systems (BRACIS)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Lattimore and Szepesvári, 2018] Tor Lattimore and Csaba Szepesvári. Bandit Algorithms. Cambridge University Press, 2018. preprint available at https://tor-lattimore.com/ downloads/book/book.pdf.","title":"Tor Lattimore and Csaba Szepesvári. Bandit Algorithms","authors":"Szepesvári Lattimore","year":"2018","venue":null,"doi":null,"url":"https://tor-lattimore.com/downloads/book/book.pdf","internal_url":null},{"position":8,"raw":"[Li et al., 2010] Lihong Li, Wei Chu, John Langford, and Robert E Schapire. A contextual-bandit approach to per- sonalized news article recommendation. In Proceedings of the 19th international conference on World wide web, pages 661-670. ACM, 2010.","title":"A contextual-bandit approach to personalized news article recommendation","authors":"Lihong Li, Wei Chu, John Langford, Robert E Schapire","year":"2010","venue":"Proceedings of the 19th international conference on World wide web","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Li et al., 2012] Lihong Li, Wei Chu, John Langford, Taesup Moon, and Xuanhui Wang. An unbiased offline evalua- tion of contextual bandit algorithms with generalized lin- ear models. In Proceedings of the Workshop on On-line Trading of Exploration and Exploitation 2, pages 19-36, 2012.","title":"An unbiased offline evaluation of contextual bandit algorithms with generalized linear models","authors":"Lihong Li, Wei Chu, John Langford, Taesup Moon, Xuanhui Wang","year":"2012","venue":"Proceedings of the Workshop on On-line Trading of Exploration and Exploitation 2","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li et al., 2017] Lihong Li, Yu Lu, and Dengyong Zhou. Provably optimal algorithms for generalized linear contex- tual bandits. arXiv preprint arXiv:1703.00048, 2017.","title":"Provably optimal algorithms for generalized linear contextual bandits","authors":"Lihong Li, Yu Lu, Dengyong Zhou","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Liu et al., 2011] Liwei Liu, Nikolay Mehandjiev, and Dong-Ling Xu. Multi-criteria service recommendation based on user criteria preferences. In Proceedings of the fifth ACM conference on Recommender systems, pages 77- 84. ACM, 2011.","title":"Multi-criteria service recommendation based on user criteria preferences","authors":"Liwei Liu, Nikolay Mehandjiev, Dong-Ling Xu","year":"2011","venue":"Proceedings of the fifth ACM conference on Recommender systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Liu, 2009] Tie-Yan Liu. Learning to rank for information retrieval. Found. Trends Inf. Retr., 3(3):225-331, March 2009.","title":"Learning to rank for information retrieval","authors":"Tie-Yan Liu, Liu","year":"2009","venue":"Found. Trends Inf. Retr","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mikolov et al., 2013] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed repre- sentations of words and phrases and their compositional- ity. In Advances in neural information processing systems, pages 3111-3119, 2013.","title":"Distributed representations of words and phrases and their compositionality","authors":"Mikolov","year":"2013","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Otunba et al., 2017] Rasaq Otunba, Raimi A Rufai, and Jes- sica Lin. Mpr: Multi-objective pairwise ranking. In Proceedings of the Eleventh ACM Conference on Recom- mender Systems, pages 170-178. ACM, 2017.","title":"Mpr: Multi-objective pairwise ranking","authors":"Rasaq Otunba, Raimi A Rufai, Jessica Lin","year":"2017","venue":"Proceedings of the Eleventh ACM Conference on Recommender Systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Radlinski et al., 2008] Filip Radlinski, Robert Kleinberg, and Thorsten Joachims. Learning diverse rankings with multi-armed bandits. In Proceedings of the 25th interna- tional conference on Machine learning, pages 784-791. ACM, 2008.","title":"Learning diverse rankings with multi-armed bandits","authors":"Filip Radlinski, Robert Kleinberg, Thorsten Joachims","year":"2008","venue":"Proceedings of the 25th international conference on Machine learning","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Robbins, 1985] Herbert Robbins. Some aspects of the se- quential design of experiments. In Herbert Robbins Se- lected Papers, pages 169-177. Springer, 1985.","title":"Herbert Robbins. Some aspects of the sequential design of experiments","authors":"Robbins","year":"1985","venue":"Herbert Robbins Selected Papers","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Roijers et al., 2017] Diederik M Roijers, Luisa M Zint- graf, and Ann Nowé. Interactive thompson sampling for multi-objective multi-armed bandits. In International Conference on Algorithmic DecisionTheory, pages 18-34. Springer, 2017.","title":"Interactive thompson sampling for multi-objective multi-armed bandits","authors":"Luisa M Diederik M Roijers, Ann Zintgraf, Nowé","year":"2017","venue":"International Conference on Algorithmic DecisionTheory","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Tallapally et al., 2018] Dharahas Tallapally, Rama Syamala Sreepada, Bidyut Kr Patra, and Korra Sathya Babu. User preference learning in multi-criteria recommendations us- ing stacked auto encoders. In Proceedings of the 12th ACM Conference on Recommender Systems, pages 475- 479. ACM, 2018.","title":"User preference learning in multi-criteria recommendations using stacked auto encoders","authors":"Tallapally","year":"2018","venue":"Proceedings of the 12th ACM Conference on Recommender Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Tekin and Turgay, 2017] Cem Tekin and Eralp Turgay. Multi-objective contextual multi-armed bandit prob- lem with a dominant objective. arXiv preprint arXiv:1708.05655, 2017.","title":"Multi-objective contextual multi-armed bandit problem with a dominant objective","authors":"Turgay ; ] Cem Tekin, Eralp Tekin, Turgay","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang et al., 2010] Hongning Wang, Yue Lu, and Chengx- iang Zhai. Latent aspect rating analysis on review text data: a rating regression approach. In Proceedings of the 16th ACM SIGKDD international conference on Knowl- edge discovery and data mining, pages 783-792. ACM, 2010.","title":"Latent aspect rating analysis on review text data: a rating regression approach","authors":"Wang","year":"2010","venue":"Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2011] Hongning Wang, Yue Lu, and ChengX- iang Zhai. Latent aspect rating analysis without aspect keyword supervision. In Proceedings of the 17th ACM SIGKDD international conference on Knowledge discov- ery and data mining, pages 618-626. ACM, 2011.","title":"Latent aspect rating analysis without aspect keyword supervision","authors":"Wang","year":"2011","venue":"Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wanigasekara et al., 2016] Nirandika Wanigasekara, Jenny Schmalfuss, Darren Carlson, and David S Rosenblum. A bandit approach for intelligent iot service composition across heterogeneous smart spaces. In Proceedings of the 6th International Conference on the Internet of Things, pages 121-129. ACM, 2016.","title":"A bandit approach for intelligent iot service composition across heterogeneous smart spaces","authors":"Wanigasekara","year":"2016","venue":"Proceedings of the 6th International Conference on the Internet of Things","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Watkins, 1989] Christopher John Cornish Hellaby Watkins. Learning from delayed rewards. PhD thesis, King's Col- lege, Cambridge, 1989.","title":"Learning from delayed rewards","authors":"Christopher John, Cornish Hellaby Watkins","year":"1989","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Xiang et al., 2010] Biao Xiang, Daxin Jiang, Jian Pei, Xiao- hui Sun, Enhong Chen, and Hang Li. Context-aware rank- ing in web search. In Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval, pages 451-458. ACM, 2010.","title":"Context-aware ranking in web search","authors":"Biao Xiang, Daxin Jiang, Jian Pei, Xiaohui Sun, Enhong Chen, Hang Li","year":"2010","venue":"Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yahyaa and Manderick, 2015] Saba Yahyaa and Bernard Manderick. Thompson sampling for multi-objective multi- armed bandits problem. In Proceedings, page 47. Presses universitaires de Louvain, 2015.","title":"Saba Yahyaa and Bernard Manderick. Thompson sampling for multi-objective multiarmed bandits problem","authors":"Manderick Yahyaa","year":"2015","venue":"Proceedings","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zamani et al., 2017] Hamed Zamani, Michael Bendersky, Xuanhui Wang, and Mingyang Zhang. Situational con- text for ranking in personal search. In Proceedings of the 26th International Conference on World Wide Web, pages 1531-1540. International World Wide Web Conferences Steering Committee, 2017.","title":"Situational context for ranking in personal search","authors":"Hamed Zamani, Michael Bendersky, Xuanhui Wang, Mingyang Zhang","year":"2017","venue":"Proceedings of the 26th International Conference on World Wide Web","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}