{"type":"ScholarlyArticle","title":"Impression Allocation for Combating Fraud in E-commerce Via Deep Reinforcement Learning with Action Norm Penalty","authors":[{"name":"Mengchen Zhao","slug":"mengchen-zhao","given":"Mengchen","family":"Zhao","affiliation":"School of Computer Science and Engineering , Nanyang Technological University , Singapore; Alibaba-NTU Singapore Joint Research Institute , Singapore"},{"name":"Zhao Li","slug":"zhao-li","given":"Zhao","family":"Li","affiliation":"Alibaba Group , Hangzhou , China"},{"name":"Bo An","slug":"bo-an","given":"Bo","family":"An","affiliation":"School of Computer Science and Engineering , Nanyang Technological University , Singapore"},{"name":"Haifeng Lu","slug":"haifeng-lu","given":"Haifeng","family":"Lu","affiliation":"Alibaba Group , Hangzhou , China"},{"name":"Yifan Yang","slug":"yifan-yang","given":"Yifan","family":"Yang","affiliation":"Alibaba Group , Hangzhou , China"},{"name":"Chen Chu","slug":"chen-chu","given":"Chen","family":"Chu","affiliation":"Alibaba Group , Hangzhou , China"}],"abstract":"Conducting fraud transactions has become popular among e-commerce sellers to make their products favorable to the platform and buyers, which decreases the utilization efficiency of buyer impressions and jeopardizes the business environment. Fraud detection techniques are necessary but not enough for the platform since it is impossible to recognize all the fraud transactions. In this paper, we focus on improving the platform's impression allocation mechanism to maximize its profit and reduce the sellers' fraudulent behaviors simultaneously. First, we learn a seller behavior model to predict the sellers' fraudulent behaviors from the real-world data provided by one of the largest e-commerce company in the world. Then, we formulate the platform's impression allocation problem as a continuous Markov Decision Process (MDP) with unbounded action space. In order to make the action executable in practice and facilitate learning, we propose a novel deep reinforcement learning algorithm DDPG-ANP that introduces an action norm penalty to the reward function. Experimental results show that our algorithm significantly outperforms existing baselines in terms of scalability and solution quality.","doi":"10.24963/ijcai.2018/548","url":"https://ijcai.org/proceedings/2018/548","pdf":"https://ijcai.org/proceedings/2018/0548.pdf","markdown":"https://ijcai.org/proceedings/2018/548.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/548","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Multidisciplinary Topics and Applications","track":"Main track","number":548,"pages":{"first":3940,"last":3946},"keywords":["Machine Learning Applications: Applications of Reinforcement Learning","Multidisciplinary Topics and Applications: Multidisciplinary Topics and Applications"],"videos":[],"references":[{"position":1,"raw":"[Cai et al., 2018a] Qingpeng Cai, Aris Filos-Ratsikas, Pingzhong Tang, and Yiwei Zhang. Reinforcement mechanism design for e-commerce. In Proceedings of the 27th International Conference on World Wide Web, 2018.","title":"Reinforcement mechanism design for e-commerce","authors":"Cai","year":"2018","venue":"Proceedings of the 27th International Conference on World Wide Web","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cai et al., 2018b] Qingpeng Cai, Aris Filos-Ratsikas, Pingzhong Tang, and Yiwei Zhang. Reinforcement mech- anism design for fraudulent behaviour in e-commerce. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence, 2018.","title":"Reinforcement mechanism design for fraudulent behaviour in e-commerce","authors":"Cai","year":"2018","venue":"Proceedings of the 32nd AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chou et al., 2017] Po-Wei Chou, Daniel Maturana, and Se- bastian Scherer. Improving stochastic policy gradients in continuous control with deep reinforcement learning using the beta distribution. In Proceedings of the International Conference on Machine Learning, pages 834-843, 2017.","title":"Improving stochastic policy gradients in continuous control with deep reinforcement learning using the beta distribution","authors":"Po-Wei Chou, Daniel Maturana, Sebastian Scherer","year":"2017","venue":"Proceedings of the International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Hooi et al., 2016] Bryan Hooi, Hyun Ah Song, Alex Beutel, Neil Shah, Kijung Shin, and Christos Faloutsos. Fraudar: Bounding graph fraud in the face of camouflage. In Pro- ceedings of the 22nd ACM SIGKDD International Con- ference on Knowledge Discovery and Data Mining, pages 895-904, 2016.","title":"Fraudar: Bounding graph fraud in the face of camouflage","authors":"Bryan Hooi, Hyun Ah Song, Alex Beutel, Neil Shah, Kijung Shin, Christos Faloutsos","year":"2016","venue":"Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Juan et al., 2016] Yuchin Juan, Yong Zhuang, Wei-Sheng Chin, and Chih-Jen Lin. Field-aware factorization ma- chines for ctr prediction. In Proceedings of the 10th ACM Conference on Recommender Systems, pages 43-50, 2016.","title":"Field-aware factorization machines for ctr prediction","authors":"Juan","year":"2016","venue":"Proceedings of the 10th ACM Conference on Recommender Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Lillicrap et al., 2015] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous con- trol with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015.","title":"Continuous control with deep reinforcement learning","authors":"Jonathan J Timothy P Lillicrap, Alexander Hunt, Nicolas Pritzel, Tom Heess, Yuval Erez, David Tassa, Daan Silver, Wierstra","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Lim et al., 2010] Ee-Peng Lim, Viet-An Nguyen, Nitin Jin- dal, Bing Liu, and Hady Wirawan Lauw. Detecting prod- uct review spammers using rating behaviors. In Proceed- ings of the 19th ACM International Conference on In- formation and Knowledge Management, pages 939-948, 2010.","title":"Detecting product review spammers using rating behaviors","authors":"Ee-Peng Lim, Viet-An Nguyen, Nitin Jindal, Bing Liu, Hady Wirawan Lauw","year":"2010","venue":"Proceedings of the 19th ACM International Conference on Information and Knowledge Management","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Mao et al., 2015] Renxin Mao, Zhao Li, and Jinhua Fu. Fraud transaction recognition: A money flow network ap- proach. In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management, pages 1871-1874, 2015.","title":"Fraud transaction recognition: A money flow network approach","authors":"Mao","year":"2015","venue":"Proceedings of the 24th ACM International on Conference on Information and Knowledge Management","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidje- land, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Pedregosa et al., 2011] Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit-learn: Machine learning in python. Journal of Machine Learning Re- search, 12(Oct):2825-2830, 2011.","title":"Scikit-learn: Machine learning in python","authors":"Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg","year":"2011","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Shen et al., 2017] Weiran Shen, Binghui Peng, Hanpeng Liu, Michael Zhang, Ruohan Qian, Yan Hong, Zhi Guo, Zongyao Ding, Pengjun Lu, and Pingzhong Tang. Re- inforcement mechanism design, with applications to dy- namic pricing in sponsored search auctions. arXiv preprint arXiv:1711.10279, 2017.","title":"Reinforcement mechanism design, with applications to dynamic pricing in sponsored search auctions","authors":"Shen","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Silver et al., 2014] David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International Conference on Machine Learning, pages 387-395, 2014.","title":"Deterministic policy gradient algorithms","authors":"David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, Martin Riedmiller","year":"2014","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Silver et al., 2016] David Silver, Aja Huang, Chris J Maddi- son, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Pan- neershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484-489, 2016.","title":"Mastering the game of go with deep neural networks and tree search","authors":"David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den, Marc Driessche, Lanctot","year":"2016","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Silver et al., 2017] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676):354, 2017.","title":"Mastering the game of go without human knowledge","authors":"David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton","year":"2017","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Sismeiro and Bucklin, 2004] Catarina Sismeiro and Ran- dolph E Bucklin. Modeling purchase behavior at an e- commerce web site: A task-completion approach. Journal of marketing research, 41(3):306-323, 2004.","title":"Catarina Sismeiro and Randolph E Bucklin. Modeling purchase behavior at an ecommerce web site: A task-completion approach","authors":"Bucklin Sismeiro","year":"2004","venue":"Journal of marketing research","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Tang, 2017] Pingzhong Tang. Reinforcement mechanism design. In Proceedings of the 26th International Joint Conference on Artificial Intelligence, pages 5146-5150, 2017.","title":"Reinforcement mechanism design","authors":"Pingzhong Tang, Tang","year":"2017","venue":"Proceedings of the 26th International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2017/739"},{"position":17,"raw":"[Van den Poel and Buckinx, 2005] Dirk Van den Poel and Wouter Buckinx. Predicting online-purchasing behaviour. European Journal of Operational Research, 166(2):557- 575, 2005.","title":"Predicting online-purchasing behaviour","authors":"Poel Van Den, Dirk Buckinx, Wouter Van Den Poel, Buckinx","year":"2005","venue":"European Journal of Operational Research","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Van Hasselt et al., 2016] Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with dou- ble q-learning. In AAAI, pages 2094-2100, 2016.","title":"Deep reinforcement learning with double q-learning","authors":"Van Hasselt","year":"2016","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wang et al., 2016] Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, and Nando Freitas. Duel- ing network architectures for deep reinforcement learning. In International Conference on Machine Learning, pages 1995-2003, 2016.","title":"Dueling network architectures for deep reinforcement learning","authors":"Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, Nando Freitas","year":"2016","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Xu and Zhang, 2015] Chang Xu and Jie Zhang. Towards collusive fraud detection in online reviews. In Proceed- ings of the IEEE International Conference on Data Min- ing, pages 1051-1056, 2015.","title":"Towards collusive fraud detection in online reviews","authors":"Chang Xu, Jie Zhang","year":"2015","venue":"Proceedings of the IEEE International Conference on Data Mining","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Zhao et al., 2017] Mengchen Zhao, Bo An, Wei Gao, and Teng Zhang. Efficient label contamination attacks against black-box learning models. In Proceedings of the 26th International Joint Conference on Artificial Intelligence, pages 3945-3951, 2017.","title":"Efficient label contamination attacks against black-box learning models","authors":"Zhao","year":"2017","venue":"Proceedings of the 26th International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2017/551"},{"position":22,"raw":"[Zhao et al., 2018] Mengchen Zhao, Bo An, Yaodong Yu, Sulin Liu, and Sinno Jialin Pan. Data poisoning attacks on multi-task relationship learning. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence, pages 2628-2635, 2018.","title":"Data poisoning attacks on multi-task relationship learning","authors":"Zhao","year":"2018","venue":"Proceedings of the 32nd AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}