{"type":"ScholarlyArticle","title":"Reinforced Negative Sampling for Recommendation with Exposure Data","authors":[{"name":"Jingtao Ding","slug":"jingtao-ding","given":"Jingtao","family":"Ding","affiliation":"Beijing National Research Center for Information Science and Technology (BNRist) ,"},{"name":"Yuhan Quan","slug":"yuhan-quan","given":"Yuhan","family":"Quan","affiliation":"Beijing National Research Center for Information Science and Technology (BNRist) ,"},{"name":"Xiangnan He","slug":"xiangnan-he","given":"Xiangnan","family":"He","affiliation":"School of Information Science and Technology , University of Science and Technology of China; Department of Electronic Engineering , Tsinghua University"},{"name":"Yong Li","slug":"yong-li","given":"Yong","family":"Li","affiliation":"Beijing National Research Center for Information Science and Technology (BNRist) ,"},{"name":"Depeng Jin","slug":"depeng-jin","given":"Depeng","family":"Jin","affiliation":"Beijing National Research Center for Information Science and Technology (BNRist) ,"}],"abstract":"In implicit feedback-based recommender systems, user exposure data, which record whether or not a recommended item has been interacted by a user, provide an important clue on selecting negative training samples. In this work, we improve the negative sampler by integrating the exposure data. We propose to generate high-quality negative instances by adversarial training to favour the difficult instances, and by optimizing additional objective to favour the real negatives in exposure data. However, this idea is non-trivial to implement since the distribution of exposure data is latent and the item space is discrete. To this end, we design a novel RNS method (short for Reinforced Negative Sampler) that generates exposure-alike negative instances through feature matching technique instead of directly choosing from exposure data. Optimized under the reinforcement learning framework, RNS is able to integrate user preference signals in exposure data and hard negatives. Extensive experiments on two real-world datasets demonstrate the effectiveness and rationality of our RNS method. Our implementation is available at: https://github. com/dingjingtao/ReinforceNS.","doi":"10.24963/ijcai.2019/309","url":"https://ijcai.org/proceedings/2019/309","pdf":"https://ijcai.org/proceedings/2019/0309.pdf","markdown":"https://ijcai.org/proceedings/2019/309.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/309","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning A-L","track":"Main track","number":309,"pages":{"first":2230,"last":2236},"keywords":["Machine Learning: Recommender Systems","Machine Learning: Learning Preferences or Rankings"],"videos":[],"references":[{"position":1,"raw":"[Bayer et al., 2017] Immanuel Bayer, Xiangnan He, Bhargav Kanagal, and Steffen Rendle. A generic coordinate de- scent framework for learning from implicit feedback. In WWW, 2017.","title":"A generic coordinate descent framework for learning from implicit feedback","authors":"Bayer","year":"2017","venue":"WWW","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cai and Wang, 2018] Liwei Cai and William Yang Wang. Kbgan: Adversarial learning for knowledge graph embed- dings. In NAACL, 2018.","title":"Kbgan: Adversarial learning for knowledge graph embeddings","authors":"Liwei Cai, William Yang, Wang","year":"2018","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Caselles-Dupré et al., 2018] H. Caselles-Dupré, F. Lesaint, and J. Royo-Letelier. Word2vec applied to recommenda- tion: Hyperparameters matter. In RecSys, 2018.","title":"Word2vec applied to recommendation: Hyperparameters matter","authors":"H Caselles-Dupré, F Lesaint, J Royo-Letelier","year":"2018","venue":"RecSys","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chae et al., 2018] Dong-Kyu Chae, Jin-Soo Kang, Sang- Wook Kim, and Jung-Tae Lee. Cfgan: A generic collabo- rative filtering framework based on generative adversarial networks. In CIKM, 2018.","title":"Cfgan: A generic collaborative filtering framework based on generative adversarial networks","authors":"Dong-Kyu Chae, Jin-Soo Kang, Sang-Wook Kim, Jung-Tae Lee","year":"2018","venue":"CIKM","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2019] Jiawei Chen, Can Wang, Sheng Zhou, Qihao Shi, Yan Feng, and Chun Chen. Samwalker: So- cial recommendation with informative sampling strategy. In WWW, 2019.","title":"Samwalker: Social recommendation with informative sampling strategy","authors":"Jiawei Chen, Can Wang, Sheng Zhou, Qihao Shi, Yan Feng, Chun Chen","year":"2019","venue":"WWW","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Ding et al., 2018a] Jingtao Ding, Fuli Feng, Xiangnan He, Guanghui Yu, Yong Li, and Depeng Jin. An improved sampler for bayesian personalized ranking by leveraging view data. In WWW Companion, 2018.","title":"An improved sampler for bayesian personalized ranking by leveraging view data","authors":"Ding","year":"2018","venue":"WWW Companion","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Ding et al., 2018b] Jingtao Ding, Guanghui Yu, Xiangnan He, Yuhan Quan, Yong Li, Tat-Seng Chua, Depeng Jin, and Jiajie Yu. Improving implicit recommender systems with view data. In IJCAI, 2018.","title":"Improving implicit recommender systems with view data","authors":"Ding","year":"2018","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2018/464"},{"position":8,"raw":"[Gao et al., 2019] Chen Gao, Xiangnan He, Dahua Gan, Xi- angning Chen, Fuli Feng, Yong Li, Tat-Seng Chua, and Depeng Jin. Neural multi-task recommendation from multi-behavior data. In ICDE, 2019.","title":"Neural multi-task recommendation from multi-behavior data","authors":"Chen Gao, Xiangnan He, Dahua Gan, Xiangning Chen, Fuli Feng, Yong Li, Tat-Seng Chua, Depeng Jin","year":"2019","venue":"ICDE","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Goodfellow et al., 2014] Ian Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Gen- erative adversarial nets. In NIPS, 2014.","title":"Generative adversarial nets","authors":"Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[He et al., 2017] Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. Neural collabo- rative filtering. In WWW, 2017.","title":"Neural collaborative filtering","authors":"He","year":"2017","venue":"WWW","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[He et al., 2018] Xiangnan He, Zhankui He, Xiaoyu Du, and Tat-Seng Chua. Adversarial personalized ranking for rec- ommendation. In SIGIR, 2018.","title":"Adversarial personalized ranking for recommendation","authors":"He","year":"2018","venue":"SIGIR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Jiang et al., 2018] Zhengshen Jiang, Hongzhi Liu, Bin Fu, Zhonghai Wu, and Tao Zhang. Recommendation in het- erogeneous information networks based on generalized random walk model and bayesian personalized ranking. In WSDM, 2018.","title":"Recommendation in heterogeneous information networks based on generalized random walk model and bayesian personalized ranking","authors":"Jiang","year":"2018","venue":"WSDM","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lee et al., 2014] Pei Lee, Laks VS Lakshmanan, Mitul Ti- wari, and Sam Shah. Modeling impression discounting in large-scale recommender systems. In KDD, 2014.","title":"Modeling impression discounting in large-scale recommender systems","authors":"Pei Lee, Laks Vs Lakshmanan, Mitul Tiwari, Sam Shah","year":"2014","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2015] Yujia Li, Kevin Swersky, and Rich Zemel. Generative moment matching networks. In ICML, 2015.","title":"Generative moment matching networks","authors":"Yujia Li, Kevin Swersky, Rich Zemel","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lian et al., 2017] Jianxun Lian, Fuzheng Zhang, Min Hou, Hongwei Wang, Xing Xie, and Guangzhong Sun. Practical lessons for job recommendations in the cold-start scenario. In Recsys Challenge, 2017.","title":"Practical lessons for job recommendations in the cold-start scenario","authors":"Lian","year":"2017","venue":"Recsys Challenge","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liang et al., 2016] Dawen Liang, Laurent Charlin, James McInerney, and David M Blei. Modeling user exposure in recommendation. In WWW, 2016.","title":"Modeling user exposure in recommendation","authors":"Dawen Liang, Laurent Charlin, James Mcinerney, David M Blei","year":"2016","venue":"WWW","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lin et al., 2019] Tzu-Heng Lin, Chen Gao, and Yong Li. Cross: Cross-platform recommendation for social e- commerce. In SIGIR, 2019.","title":"Cross: Cross-platform recommendation for social ecommerce","authors":"Lin","year":"2019","venue":"SIGIR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Pan et al., 2008] Rong Pan, Yunhong Zhou, Bin Cao, Nathan N Liu, Rajan Lukose, Martin Scholz, and Qiang Yang. One-class collaborative filtering. In ICDM, 2008.","title":"One-class collaborative filtering","authors":"Rong Pan, Yunhong Zhou, Bin Cao, Nathan N Liu, Rajan Lukose, Martin Scholz, Qiang Yang","year":"2008","venue":"ICDM","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Park and Chang, 2019] Dae Hoon Park and Yi Chang. Ad- versarial sampling and training for semi-supervised infor- mation retrieval. In WWW, 2019.","title":"Adversarial sampling and training for semi-supervised information retrieval","authors":"Chang ; ] Park, Hoon Dae, Yi Park, Chang","year":"2019","venue":"WWW","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Rendle and Freudenthaler, 2014] Steffen Rendle and Christoph Freudenthaler. Improving pairwise learning for item recommendation from implicit feedback. In WSDM, 2014.","title":"Improving pairwise learning for item recommendation from implicit feedback","authors":"Steffen Rendle, Christoph Freudenthaler","year":"2014","venue":"WSDM","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Rendle et al., 2009] Steffen Rendle, Christoph Freuden- thaler, Zeno Gantner, and Lars Schmidt-Thieme. Bpr: Bayesian personalized ranking from implicit feedback. In UAI, 2009.","title":"Bpr: Bayesian personalized ranking from implicit feedback","authors":"Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, Lars Schmidt-Thieme","year":"2009","venue":"UAI","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sutton et al., 2000] Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approx- imation. In NIPS, 2000.","title":"Policy gradient methods for reinforcement learning with function approximation","authors":"David A Richard S Sutton, Mcallester, P Satinder, Yishay Singh, Mansour","year":"2000","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang et al., 2017] Jun Wang, Lantao Yu, Weinan Zhang, Yu Gong, Yinghui Xu, Benyou Wang, Peng Zhang, and Dell Zhang. Irgan: A minimax game for unifying gener- ative and discriminative information retrieval models. In SIGIR, 2017.","title":"Irgan: A minimax game for unifying generative and discriminative information retrieval models","authors":"Jun Wang, Lantao Yu, Weinan Zhang, Yu Gong, Yinghui Xu, Benyou Wang, Peng Zhang, Dell Zhang","year":"2017","venue":"SIGIR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yang et al., 2018] Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Un- biased offline recommender evaluation for missing-not-at- random implicit feedback. In RecSys, 2018.","title":"Unbiased offline recommender evaluation for missing-not-atrandom implicit feedback","authors":"Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, Deborah Estrin","year":"2018","venue":"RecSys","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhang et al., 2013] Weinan Zhang, Tianqi Chen, Jun Wang, and Yong Yu. Optimizing top-n collaborative filtering via dynamic negative item sampling. In SIGIR, 2013.","title":"Optimizing top-n collaborative filtering via dynamic negative item sampling","authors":"Weinan Zhang, Tianqi Chen, Jun Wang, Yong Yu","year":"2013","venue":"SIGIR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhang et al., 2017] Yizhe Zhang, Zhe Gan, Kai Fan, Zhi Chen, Ricardo Henao, Dinghan Shen, and Lawrence Carin. Adversarial feature matching for text generation. In ICML, 2017.","title":"Adversarial feature matching for text generation","authors":"Zhang","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2019a] Yongqi Zhang, Quanming Yao, Yingxia Shao, and Lei Chen. Nscaching: Simple and effi- cient negative sampling for knowledge graph embedding. In ICDE, 2019.","title":"Nscaching: Simple and efficient negative sampling for knowledge graph embedding","authors":"Zhang","year":"2019","venue":"ICDE","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhang et al., 2019b] Zhiqian Zhang, Chenliang Li, Zhiyong Wu, Aixin Sun, Dengpan Ye, and Xiangyang Luo. Next: a neural network framework for next poi recommendation. Frontiers of Computer Science, 2019.","title":"Next: a neural network framework for next poi recommendation","authors":"Zhang","year":"2019","venue":"Frontiers of Computer Science","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhao et al., 2018] Qian Zhao, Martijn C Willemsen, Ged- iminas Adomavicius, F Maxwell Harper, and Joseph A Konstan. Interpreting user inaction in recommender sys- tems. In RecSys, 2018.","title":"Interpreting user inaction in recommender systems","authors":"Qian Zhao, Gediminas Martijn C Willemsen, Maxwell Adomavicius, Joseph A Harper, Konstan","year":"2018","venue":"RecSys","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}