{"type":"ScholarlyArticle","title":"Beyond the Click-Through Rate: Web Link Selection with Multi-level Feedback","authors":[{"name":"Kun Chen","slug":"kun-chen","given":"Kun","family":"Chen","affiliation":""},{"name":"Kechao Cai","slug":"kechao-cai","given":"Kechao","family":"Cai","affiliation":"† Department of Computer Science and Engineering , The Chinese University of Hong Kong"},{"name":"Longbo Huang","slug":"longbo-huang","given":"Longbo","family":"Huang","affiliation":"‡ Institute for Interdisciplinary Information Sciences (IIIS) , Tsinghua University"},{"name":"John C.S. Lui","slug":"john-c-s-lui","given":"John C.S.","family":"Lui","affiliation":"† Department of Computer Science and Engineering , The Chinese University of Hong Kong"}],"abstract":"The web link selection problem is to select a small subset of web links from a large web link pool, and to place the selected links on a web page that can only accommodate a limited number of links, e.g., advertisements, recommendations, or news feeds. Despite the long concerned click-through rate which reflects the attractiveness of the link itself, revenue can only be obtained from user actions after clicks, e.g., purchasing after being directed to the product pages by recommendation links. Thus, web links have an intrinsic multi-level feedback structure. With this observation, we consider the context-free web link selection problem, where the objective is to maximize revenue while ensuring that the attractiveness is no less than a preset threshold. The key challenge of the problem is that each link's multi-level feedbacks are stochastic, and unobservable unless the link is selected. We model this problem with a constrained stochastic multi-armed bandit formulation, and design an efficient link selection algorithm, called Constrained Upper Confidence Bound algorithm (Con-UCB). We prove O(sqrt(T ln(T))) bounds on both regret and violation of the attractiveness constraint. We also conduct extensive experiments on three real-world datasets, and show that Con-UCB outperforms state-of-the-art context-free bandit algorithms concerning the multi-level feedback structure.","doi":"10.24963/ijcai.2018/459","url":"https://ijcai.org/proceedings/2018/459","pdf":"https://ijcai.org/proceedings/2018/0459.pdf","markdown":"https://ijcai.org/proceedings/2018/459.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/459","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning Applications","track":"Main track","number":459,"pages":{"first":3308,"last":3314},"keywords":["Machine Learning Applications: Applications of Reinforcement Learning","Multidisciplinary Topics and Applications: AI and the Web"],"videos":[],"references":[{"position":1,"raw":"[Aggarwal et al., 2010] Gaurav Aggarwal, Elie Bursztein, Collin Jackson, and Dan Boneh. An analysis of private browsing modes in modern browsers. In Proceedings of the 19th USENIX conference on Security, 2010.","title":"An analysis of private browsing modes in modern browsers","authors":"Gaurav Aggarwal, Elie Bursztein, Collin Jackson, Dan Boneh","year":"2010","venue":"Proceedings of the 19th USENIX conference on Security","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Agrawal and Devanur, 2014] Shipra Agrawal and Nikhil R. Devanur. Bandits with concave rewards and convex knap- sacks. In Proceedings of the Fifteenth ACM Conference on Economics and Computation, 2014.","title":"Bandits with concave rewards and convex knapsacks","authors":"Shipra Agrawal, Nikhil R Devanur","year":"2014","venue":"Proceedings of the Fifteenth ACM Conference on Economics and Computation","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Anantharam et al., 1987] Venkatachalam Anantharam, Pravin Varaiya, and Jean Walrand. Asymptotically effi- cient allocation rules for the multiarmed bandit problem with multiple plays-part i: Iid rewards. IEEE Transactions on Automatic Control, 32(11):968-976, 1987.","title":"Asymptotically efficient allocation rules for the multiarmed bandit problem with multiple plays-part i: Iid rewards","authors":"Pravin Venkatachalam Anantharam, Jean Varaiya, Walrand","year":"1987","venue":"IEEE Transactions on Automatic Control","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Auer et al., 2002] Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning, 47(2-3):235-256, 2002.","title":"Finite-time analysis of the multiarmed bandit problem","authors":"Auer","year":"2002","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Azuma, 1967] Kazuoki Azuma. Weighted sums of certain dependent random variables. Tohoku Mathematical Jour- nal, 19:357-367, 1967.","title":"Weighted sums of certain dependent random variables","authors":"Kazuoki Azuma, Azuma","year":"1967","venue":"Tohoku Mathematical Journal","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Badanidiyuru et al., 2013] Ashwinkumar Badanidiyuru, Robert Kleinberg, and Aleksandrs Slivkins. Bandits with knapsacks. In Proceedings of FOCS, 2013.","title":"Ashwinkumar Badanidiyuru, Robert Kleinberg, and Aleksandrs Slivkins. Bandits with knapsacks","authors":"Badanidiyuru","year":"2013","venue":"Proceedings of FOCS","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Bresler et al., 2016] Guy Bresler, Devavrat Shah, and Luis Filipe Voloch. Collaborative filtering with low regret. In Proceedings of ACM SIGMETRICS, 2016.","title":"Collaborative filtering with low regret","authors":"Bresler","year":"2016","venue":"Proceedings of ACM SIGMETRICS","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Bubeck et al., 2012] Sébastien Bubeck, Nicolo Cesa- Bianchi, et al. Regret analysis of stochastic and nonstochastic multi-armed bandit problems. Foundations and Trends R in Machine Learning, 5(1):1-122, 2012.","title":"Regret analysis of stochastic and nonstochastic multi-armed bandit problems","authors":"Sébastien Bubeck, Nicolo Cesa-Bianchi","year":"2012","venue":"Foundations and Trends R in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Cai et al., 2017] Kechao Cai, Kun Chen, Longbo Huang, and John C. S. Lui. Multi-level feedback web links selec- tion problem: Learning and optimization. In Proceedings of ICDM, 2017.","title":"Multi-level feedback web links selection problem: Learning and optimization","authors":"Cai","year":"2017","venue":"Proceedings of ICDM","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Chen et al., 2013] Wei Chen, Yajun Wang, and Yang Yuan. Combinatorial multi-armed bandit: General framework, results and applications. In ICML, 2013.","title":"Combinatorial multi-armed bandit: General framework, results and applications","authors":"Wei Chen, Yajun Wang, Yang Yuan","year":"2013","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Chen et al., 2018] Kun Chen, Kechao Cai, Longbo Huang, and John C. S. Lui. Beyond the click-through rate: Web link selection with multi-level feedback. arXiv preprint arXiv:1805.01702, 2018.","title":"Beyond the click-through rate: Web link selection with multi-level feedback","authors":"Kun Chen, Kechao Cai, Longbo Huang, John C S Lui","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Chuang and Ho, 2016] Isaac Chuang and Andrew Dean Ho. Harvardx and mitx: Four years of open online courses-fall 2012-summer 2016. SSRN, 2016.","title":"Harvardx and mitx: Four years of open online courses-fall 2012-summer","authors":"Chuang, Ho, Isaac Chuang, Andrew Dean Ho","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Deng et al., 2017] Alex Deng, Jiannan Lu, and Jonthan Litz. Trustworthy analysis of online a/b tests: Pitfalls, challenges and solutions. In Proceedings of WSDM, 2017.","title":"Trustworthy analysis of online a/b tests: Pitfalls, challenges and solutions","authors":"Alex Deng, Jiannan Lu, Jonthan Litz","year":"2017","venue":"Proceedings of WSDM","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Ding et al., 2013] Wenkui Ding, Tao Qin, Xu-Dong Zhang, and Tie-Yan Liu. Multi-armed bandit with budget con- straint and variable costs. In AAAI, 2013.","title":"Multi-armed bandit with budget constraint and variable costs","authors":"Ding","year":"2013","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Elahi et al., 2016] Mehdi Elahi, Francesco Ricci, and Neil Rubens. A survey of active learning in collaborative fil- tering recommender systems. Computer Science Review, 20:29-50, 2016.","title":"A survey of active learning in collaborative filtering recommender systems","authors":"Mehdi Elahi, Francesco Ricci, Neil Rubens","year":"2016","venue":"Computer Science Review","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Gandhi et al., 2006] Rajiv Gandhi, Samir Khuller, Srini- vasan Parthasarathy, and Aravind Srinivasan. Dependent rounding and its applications to approximation algorithms. Journal of the ACM (JACM), 53(3):324-360, 2006.","title":"Dependent rounding and its applications to approximation algorithms","authors":"Rajiv Gandhi, Samir Khuller, Srinivasan Parthasarathy, Aravind Srinivasan","year":"2006","venue":"Journal of the ACM (JACM)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kaggle, 2015] Kaggle. Avito context ad clicks, 2015. https://www.kaggle.com/c/avito-context-ad-clicks. [Kaggle, 2016] Kaggle. Coupon purchase data, 2016. https://www.kaggle.com/c/coupon-purchase-prediction.","title":"Kaggle. Avito context ad clicks","authors":"Kaggle","year":"2015","venue":null,"doi":null,"url":"https://www.kaggle.com/c/coupon-purchase-prediction","internal_url":null},{"position":18,"raw":"[Kleinberg et al., 2008] Robert Kleinberg, Aleksandrs Slivkins, and Eli Upfal. Multi-armed bandits in metric spaces. In Proceedings of STOC, 2008.","title":"Multi-armed bandits in metric spaces","authors":"Robert Kleinberg, Aleksandrs Slivkins, Eli Upfal","year":"2008","venue":"Proceedings of STOC","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Kohavi et al., 2014] Ron Kohavi, Alex Deng, Roger Long- botham, and Ya Xu. Seven rules of thumb for web site experimenters. In Proceedings of SIGKDD, 2014.","title":"Seven rules of thumb for web site experimenters","authors":"Ron Kohavi, Alex Deng, Roger Longbotham, Ya Xu","year":"2014","venue":"Proceedings of SIGKDD","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Komiyama et al., 2015] Junpei Komiyama, Junya Honda, and Hiroshi Nakagawa. Optimal regret analysis of thomp- son sampling in stochastic multi-armed bandit problem with multiple plays. In ICML, 2015.","title":"Optimal regret analysis of thompson sampling in stochastic multi-armed bandit problem with multiple plays","authors":"Komiyama","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Kumar, 2015] Subodha Kumar. Optimization Issues in Web and Mobile Advertising: Past and Future Trends. Springer, 2015.","title":"Subodha Kumar. Optimization Issues in Web and Mobile Advertising: Past and Future Trends","authors":"Kumar","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Lagrée et al., 2016] Paul Lagrée, Claire Vernade, and Olivier Cappe. Multiple-play bandits in the position-based model. In NIPS, 2016.","title":"Multiple-play bandits in the position-based model","authors":"Paul Lagrée, Claire Vernade, Olivier Cappe","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Langheinrich et al., 1999] Marc Langheinrich, Atsuyoshi Nakamura, Naoki Abe, Tomonari Kamba, and Yoshiyuki Koseki. Unintrusive customization techniques for web ad- vertising. Computer Networks, 31(11):1259-1272, 1999.","title":"Unintrusive customization techniques for web advertising","authors":"Marc Langheinrich, Atsuyoshi Nakamura, Naoki Abe, Tomonari Kamba, Yoshiyuki Koseki","year":"1999","venue":"Computer Networks","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Li et al., 2010] Lihong Li, Wei Chu, John Langford, and Robert E. Schapire. A contextual-bandit approach to per- sonalized news article recommendation. In Proceedings of WWW, 2010.","title":"A contextual-bandit approach to personalized news article recommendation","authors":"Lihong Li, Wei Chu, John Langford, Robert E Schapire","year":"2010","venue":"Proceedings of WWW","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Li et al., 2016] Shuai Li, Alexandros Karatzoglou, and Claudio Gentile. Collaborative filtering bandits. In Pro- ceedings of SIGIR, 2016.","title":"Collaborative filtering bandits","authors":"Shuai Li, Alexandros Karatzoglou, Claudio Gentile","year":"2016","venue":"Proceedings of SIGIR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Locatelli et al., 2016] Andrea Locatelli, Maurilio Gutzeit, and Alexandra Carpentier. An optimal algorithm for the thresholding bandit problem. In ICML, 2016.","title":"An optimal algorithm for the thresholding bandit problem","authors":"Andrea Locatelli, Maurilio Gutzeit, Alexandra Carpentier","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Lohtia et al., 2003] Ritu Lohtia, Naveen Donthu, and Ed- mund K Hershberger. The impact of content and design elements on banner advertising click-through rates. Jour- nal of advertising Research, 43(4):410-418, 2003.","title":"The impact of content and design elements on banner advertising click-through rates","authors":"Ritu Lohtia, Naveen Donthu, Edmund K Hershberger","year":"2003","venue":"Journal of advertising Research","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Meng et al., 2016] Wei Meng, Byoungyoung Lee, Xinyu Xing, and Wenke Lee. Trackmeornot: Enabling flexible control on web tracking. In Proceedings of WWW, 2016.","title":"Trackmeornot: Enabling flexible control on web tracking","authors":"Wei Meng, Byoungyoung Lee, Xinyu Xing, Wenke Lee","year":"2016","venue":"Proceedings of WWW","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Mookerjee et al., 2016] Radha Mookerjee, Subodha Ku- mar, and Vijay S Mookerjee. Optimizing performance- based internet advertisement campaigns. Operations Re- search, 65(1):38-54, 2016.","title":"Optimizing performancebased internet advertisement campaigns","authors":"Radha Mookerjee, Subodha Kumar, Vijay S Mookerjee","year":"2016","venue":"Operations Research","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Pivazyan, 2004] Karen Arman Pivazyan. Decision making in multi-agent systems. Stanford University, 2004.","title":"Decision making in multi-agent systems","authors":"Karen Arman, Pivazyan","year":"2004","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Spooner, 2014] Jason Spooner. Why cost per acqui- sition is the only metric that really matters, 2014. https://socialmediaexplorer.com.","title":"Jason Spooner. Why cost per acquisition is the only metric that really matters","authors":"Spooner","year":"2014","venue":null,"doi":null,"url":"https://socialmediaexplorer.com","internal_url":null},{"position":32,"raw":"[Uchiya et al., 2010] Taishi Uchiya, Atsuyoshi Nakamura, and Mineichi Kudo. Algorithms for adversarial bandit problems with multiple plays. In International Conference on Algorithmic Learning Theory, 2010.","title":"Algorithms for adversarial bandit problems with multiple plays","authors":"Taishi Uchiya, Atsuyoshi Nakamura, Mineichi Kudo","year":"2010","venue":"International Conference on Algorithmic Learning Theory","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Wu et al., 2015] Huasen Wu, R. Srikant, Xin Liu, and Chong Jiang. Algorithms with logarithmic or sublinear regret for constrained contextual bandits. In NIPS, 2015.","title":"Algorithms with logarithmic or sublinear regret for constrained contextual bandits","authors":"Huasen Wu, R Srikant, Xin Liu, Chong Jiang","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Xia et al., 2016] Yingce Xia, Tao Qin, Weidong Ma, Neng- hai Yu, and Tie-Yan Liu. Budgeted multi-armed bandits with multiple plays. In Proceedings of IJCAI, 2016.","title":"Budgeted multi-armed bandits with multiple plays","authors":"Yingce Xia, Tao Qin, Weidong Ma, Nenghai Yu, Tie-Yan Liu","year":"2016","venue":"Proceedings of IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2016/315"},{"position":35,"raw":"[Xu et al., 2015] Ya Xu, Nanyu Chen, Addrian Fernandez, Omar Sinno, and Anmol Bhasin. From infrastructure to culture: A/b testing challenges in large scale social net- works. In Proceedings of SIGKDD, 2015.","title":"From infrastructure to culture: A/b testing challenges in large scale social networks","authors":"Ya Xu, Nanyu Chen, Addrian Fernandez, Omar Sinno, Anmol Bhasin","year":"2015","venue":"Proceedings of SIGKDD","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}