{"type":"ScholarlyArticle","title":"A Dual Input-aware Factorization Machine for CTR Prediction","authors":[{"name":"Wantong Lu","slug":"wantong-lu","given":"Wantong","family":"Lu","affiliation":"Shenzhen International Graduate School , Tsinghua University 2 Data 61 CSIRO , Sydney , Australia"},{"name":"Yantao Yu","slug":"yantao-yu","given":"Yantao","family":"Yu","affiliation":"Shenzhen International Graduate School , Tsinghua University 2 Data 61 CSIRO , Sydney , Australia"},{"name":"Yongzhe Chang","slug":"yongzhe-chang","given":"Yongzhe","family":"Chang","affiliation":"Shenzhen International Graduate School , Tsinghua University 2 Data 61 CSIRO , Sydney , Australia"},{"name":"Zhen Wang","slug":"zhen-wang","given":"Zhen","family":"Wang","affiliation":"UBTECH Sydney AI Centre , The University of Sydney , Australia"},{"name":"Chenhui Li","slug":"chenhui-li","given":"Chenhui","family":"Li","affiliation":"Shenzhen International Graduate School , Tsinghua University 2 Data 61 CSIRO , Sydney , Australia"},{"name":"Bo Yuan","slug":"bo-yuan","given":"Bo","family":"Yuan","affiliation":"Shenzhen International Graduate School , Tsinghua University 2 Data 61 CSIRO , Sydney , Australia"}],"abstract":"Factorization Machines (FMs) refer to a class of general predictors working with real valued feature vectors, which are well-known for their ability to estimate model parameters under significant sparsity and have found successful applications in many areas such as the click-through rate (CTR) prediction. However, standard FMs only produce a single fixed representation for each feature across different input instances, which may limit the CTR model’s expressive and predictive power. Inspired by the success of Input-aware Factorization Machines (IFMs), which aim to learn more flexible and informative representations of a given feature according to different input instances, we propose a novel model named Dual Input-aware Factorization Machines (DIFMs) that can adaptively reweight the original feature representations at the bit-wise and vector-wise levels simultaneously. Furthermore, DIFMs strategically integrate various components including Multi-Head Self-Attention, Residual Networks and DNNs into a unified end-to-end model. Comprehensive experiments on two real-world CTR prediction datasets show that the DIFM model can outperform several state-of-the-art models consistently.","doi":"10.24963/ijcai.2020/434","url":"https://ijcai.org/proceedings/2020/434","pdf":"https://ijcai.org/proceedings/2020/0434.pdf","markdown":"https://ijcai.org/proceedings/2020/434.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/434","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":434,"pages":{"first":3139,"last":3145},"keywords":["Multidisciplinary Topics and Applications: Recommender Systems","Machine Learning: Recommender Systems","Data Mining: Classification, Semi-Supervised Learning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24246/file_24246.m3u8","length_sec":296,"provider":"hls"}],"references":[{"position":1,"raw":"[Amodei et al., 2016] Dario Amodei, Sundaram Anantha- narayanan, Rishita Anubhai, Jingliang Bai, Eric Batten- berg, Carl Case, Jared Casper, Bryan Catanzaro, Qiang Cheng, Guoliang Chen, et al. Deep speech 2: End-to- end speech recognition in english and mandarin. In Inter- national conference on machine learning, pages 173-182, 2016.","title":"Deep speech 2: End-toend speech recognition in english and mandarin","authors":"Dario Amodei, Rishita Sundaram Ananthanarayanan, Jingliang Anubhai, Eric Bai, Carl Battenberg, Jared Case, Bryan Casper, Qiang Catanzaro, Guoliang Cheng, Chen","year":"2016","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cheng et al., 2016] Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al. Wide & deep learning for recom- mender systems. In Proceedings of the 1st workshop on deep learning for recommender systems, pages 7-10. ACM, 2016.","title":"Wide & deep learning for recommender systems","authors":"Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir","year":"2016","venue":"Proceedings of the 1st workshop on deep learning for recommender systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cho et al., 2014] Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078, 2014.","title":"Learning phrase representations using rnn encoder-decoder for statistical machine translation","authors":"Cho","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Guo et al., 2017] Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. Deepfm: a factorization-machine based neural network for ctr predic- tion. arXiv preprint arXiv:1703.04247, 2017.","title":"Deepfm: a factorization-machine based neural network for ctr prediction","authors":"Guo","year":"2017","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2017/239"},{"position":5,"raw":"[He and Chua, 2017] Xiangnan He and Tat-Seng Chua. Neu- ral factorization machines for sparse predictive analytics. In Proceedings of the 40th International ACM SIGIR con- ference on Research and Development in Information Re- trieval, pages 355-364. ACM, 2017.","title":"Xiangnan He and Tat-Seng Chua. Neural factorization machines for sparse predictive analytics","authors":"Chua He","year":"2017","venue":"Proceedings of the 40th International ACM SIGIR conference on Research and Development in Information Retrieval","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proceedings of the IEEE conference on CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Juan et al., 2016] Yuchin Juan, Yong Zhuang, Wei-Sheng Chin, and Chih-Jen Lin. Field-aware factorization ma- chines for ctr prediction. In Proceedings of the 10th ACM Conference on Recommender Systems, pages 43-50. ACM, 2016.","title":"Field-aware factorization machines for ctr prediction","authors":"Juan","year":"2016","venue":"Proceedings of the 10th ACM Conference on Recommender Systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kingma and Ba, 2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Lee et al., 2012] Kuang-chih Lee, Burkay Orten, Ali Das- dan, and Wentong Li. Estimating conversion rate in dis- play advertising from past erformance data. In Proceed- ings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 768-776. ACM, 2012.","title":"Estimating conversion rate in display advertising from past erformance data","authors":"Kuang-Chih Lee, Burkay Orten, Ali Dasdan, Wentong Li","year":"2012","venue":"Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lian et al., 2018] Jianxun Lian, Xiaohuan Zhou, Fuzheng Zhang, Zhongxia Chen, Xing Xie, and Guangzhong Sun. xdeepfm: Combining explicit and implicit feature inter- actions for recommender systems. In Proceedings of the 24th ACM SIGKDD International Conference on Knowl- edge Discovery & Data Mining, pages 1754-1763. ACM, 2018.","title":"xdeepfm: Combining explicit and implicit feature interactions for recommender systems","authors":"Lian","year":"2018","venue":"Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[McMahan et al., 2013] H Brendan McMahan, Gary Holt, David Sculley, Michael Young, Dietmar Ebner, Julian Grady, Lan Nie, Todd Phillips, Eugene Davydov, Daniel Golovin, et al. Ad click prediction: a view from the trenches. In Proceedings of the 19th ACM SIGKDD in- ternational conference on Knowledge discovery and data mining, pages 1222-1230. ACM, 2013.","title":"Ad click prediction: a view from the trenches","authors":"Gary H Brendan Mcmahan, David Holt, Michael Sculley, Dietmar Young, Julian Ebner, Lan Grady, Todd Nie, Eugene Phillips, Davydov, Daniel Golovin","year":"2013","venue":"Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Qu et al., 2016] Yanru Qu, Han Cai, Kan Ren, Weinan Zhang, Yong Yu, Ying Wen, and Jun Wang. Product- based neural networks for user response prediction. In 2016 IEEE 16th ICDM, pages 1149-1154. IEEE, 2016.","title":"Productbased neural networks for user response prediction","authors":"Qu","year":"2016","venue":"IEEE 16th ICDM","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Rendle, 2010] Steffen Rendle. Factorization machines. In 2010 IEEE ICDM, pages 995-1000. IEEE, 2010.","title":"Factorization machines","authors":"Steffen Rendle","year":"2010","venue":"IEEE ICDM","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Song et al., 2019] Weiping Song, Chence Shi, Zhiping Xiao, Zhijian Duan, Yewen Xu, Ming Zhang, and Jian Tang. Autoint: Automatic feature interaction learning via self-attentive neural networks. In Proceedings of the 28th CIKM, pages 1161-1170. ACM, 2019.","title":"Autoint: Automatic feature interaction learning via self-attentive neural networks","authors":"Weiping Song, Chence Shi, Zhiping Xiao, Zhijian Duan, Yewen Xu, Ming Zhang, Jian Tang","year":"2019","venue":"Proceedings of the 28th CIKM","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing sys- tems, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Wang et al., 2017] Ruoxi Wang, Bin Fu, Gang Fu, and Min- gliang Wang. Deep & cross network for ad click predic- tions. In Proceedings of the ADKDD'17, page 12. ACM, 2017.","title":"Deep & cross network for ad click predictions","authors":"Wang","year":"2017","venue":"Proceedings of the ADKDD'17","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Wen et al., 2019] Hong Wen, Jing Zhang, Quan Lin, Keping Yang, and Pipei Huang. Multi-level deep cascade trees for conversion rate prediction in recommendation system. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 33, pages 338-345, 2019.","title":"Multi-level deep cascade trees for conversion rate prediction in recommendation system","authors":"Wen","year":"2019","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Xiao et al., 2017] Jun Xiao, Hao Ye, Xiangnan He, Han- wang Zhang, Fei Wu, and Tat-Seng Chua. Atten- tional factorization machines: Learning the weight of fea- ture interactions via attention networks. arXiv preprint arXiv:1708.04617, 2017.","title":"Attentional factorization machines: Learning the weight of feature interactions via attention networks","authors":"Jun Xiao, Xiangnan Hao Ye, Hanwang He, Fei Zhang, Tat-Seng Wu, Chua","year":"2017","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2017/435"},{"position":19,"raw":"[Yu et al., 2019] Yantao Yu, Zhen Wang, and Bo Yuan. An input-aware factorization machine for sparse prediction. In Proceedings of the 28th IJCAI, pages 1466-1472. AAAI Press, 2019.","title":"An input-aware factorization machine for sparse prediction","authors":"Yu","year":"2019","venue":"Proceedings of the 28th IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2019/203"},{"position":20,"raw":"[Zhang et al., 2016] Weinan Zhang, Tianming Du, and Jun Wang. Deep learning over multi-field categorical data. In European conference on information retrieval, pages 45- 57. Springer, 2016.","title":"Deep learning over multi-field categorical data","authors":"Weinan Zhang, Tianming Du, Jun Wang","year":"2016","venue":"European conference on information retrieval","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Zhou et al., 2018] Guorui Zhou, Xiaoqiang Zhu, Chenru Song, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. Deep interest network for click- through rate prediction. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discov- ery & Data Mining, pages 1059-1068. ACM, 2018.","title":"Deep interest network for clickthrough rate prediction","authors":"Guorui Zhou, Xiaoqiang Zhu, Chenru Song, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, Kun Gai","year":"2018","venue":"Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zhou et al., 2019] Guorui Zhou, Na Mou, Ying Fan, Qi Pi, Weijie Bian, Chang Zhou, Xiaoqiang Zhu, and Kun Gai. Deep interest evolution network for click-through rate pre- diction. In Proceedings of the AAAI Conference on Artifi- cial Intelligence, volume 33, pages 5941-5948, 2019.","title":"Deep interest evolution network for click-through rate prediction","authors":"Guorui Zhou, Na Mou, Ying Fan, Qi Pi, Weijie Bian, Chang Zhou, Xiaoqiang Zhu, Kun Gai","year":"2019","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}