{"type":"ScholarlyArticle","title":"GS2P: A Generative Pre-trained Learning to Rank Model with Over-parameterization for Web-Scale Search (Extended Abstract)","authors":[{"name":"Yuchen Li","slug":"yuchen-li","given":"Yuchen","family":"Li","affiliation":"Shanghai Jiao Tong University , China; Baidu Inc. , China"},{"name":"Haoyi Xiong","slug":"haoyi-xiong","given":"Haoyi","family":"Xiong","affiliation":"Baidu Inc. , China"},{"name":"Linghe Kong","slug":"linghe-kong","given":"Linghe","family":"Kong","affiliation":"Shanghai Jiao Tong University , China"},{"name":"Jiang Bian","slug":"jiang-bian","given":"Jiang","family":"Bian","affiliation":"Baidu Inc. , China"},{"name":"Shuaiqiang Wang","slug":"shuaiqiang-wang","given":"Shuaiqiang","family":"Wang","affiliation":"Baidu Inc. , China"},{"name":"Guihai Chen","slug":"guihai-chen","given":"Guihai","family":"Chen","affiliation":"Shanghai Jiao Tong University , China"},{"name":"Dawei Yin","slug":"dawei-yin","given":"Dawei","family":"Yin","affiliation":"Baidu Inc. , China"}],"abstract":"While Learning to Rank (LTR) is widely employed in web searches to prioritize pertinent webpages from the retrieved contents based on input queries, traditional LTR models stumble over two principal stumbling blocks leading to subpar performance: 1) the lack of well-annotated query-webpage pairs with ranking scores to cover search queries of various popularity, debilitating their coverage of search queries across the popularity spectrum, and 2) ill-trained models that are incapable of inducing generalized representations for LTR, culminating in overfitting. To tackle above challenges, we proposed a Generative Semi-supervised Pre-trained (GS2P) LTR model. Specifically, GS2P first generates pseudo-labels for the unlabeled samples using tree-based LTR models after a series of co-training procedures, then learns the representations of query-webpage pairs with self-attentive transformers via both discriminative and generative losses. Finally, GS2P boosts the performance of LTR through incorporating Random Fourier Features to over-parameterize the models into \"interpolating regime\", so as to enjoy the further descent of generalization errors with learned representations. We conduct extensive offline experiments on a publicly available dataset and a real-world dataset collected from a large-scale search engine. The results show that GS2P can achieve the best performance on both datasets, compared to baselines.\r\nWe also deploy GS2P at a large-scale web search engine with realistic traffic, where we can still observe significant improvement in real-world applications.","doi":"10.24963/ijcai.2024/936","url":"https://ijcai.org/proceedings/2024/936","pdf":"https://ijcai.org/proceedings/2024/0936.pdf","markdown":"https://ijcai.org/proceedings/2024/936.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/936","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Sister Conferences Best Papers","track":"Sister Conferences Best Papers","number":936,"pages":{"first":8433,"last":8438},"keywords":["Data Mining: DM: Mining text, web, social media"],"videos":[],"references":[{"position":1,"raw":"[Belkin et al., 2019] Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine- learning practice and the classical bias-variance trade- off. Proceedings of the National Academy of Sciences, 116(32):15849-15854, 2019.","title":"Reconciling modern machinelearning practice and the classical bias-variance tradeoff","authors":"Mikhail Belkin, Daniel Hsu, Siyuan Ma, Soumik Mandal","year":"2019","venue":"Proceedings of the National Academy of Sciences","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Belkin, 2021] Mikhail Belkin. Fit without fear: remark- able mathematical phenomena of deep learning through the prism of interpolation. Acta Numerica, 30:203-248, 2021.","title":"Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolation","authors":"Mikhail Belkin, Belkin","year":"2021","venue":"Acta Numerica","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bruch et al., 2019] Sebastian Bruch, Masrour Zoghi, Michael Bendersky, and Marc Najork. Revisiting ap- proximate metric optimization in the age of deep neural networks. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1241-1244, 2019.","title":"Revisiting approximate metric optimization in the age of deep neural networks","authors":"Sebastian Bruch, Masrour Zoghi, Michael Bendersky, Marc Najork","year":"2019","venue":"Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Burges et al., 2005] Christopher J. C. Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Gregory N. Hullender. Learning to rank using gra- dient descent. In Machine Learning, Proceedings of the Twenty-Second International Conference, ICML, pages 89- 96, 2005.","title":"Learning to rank using gradient descent","authors":"J C Christopher, Tal Burges, Erin Shaked, Ari Renshaw, Matt Lazier, Nicole Deeds, Gregory N Hamilton, Hullender","year":"2005","venue":"Proceedings of the Twenty-Second International Conference, ICML","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Burges et al., 2006] Christopher J. C. Burges, Robert Ragno, and Quoc Viet Le. Learning to rank with nonsmooth cost functions. In Advances in Neural Information Processing Systems 19, Proceedings of the Twentieth Annual Confer- ence on Neural Information Processing Systems, pages 193-200, 2006.","title":"Learning to rank with nonsmooth cost functions","authors":"J C Christopher, Robert Burges, Quoc Ragno, Le Viet","year":"2006","venue":"Proceedings of the Twentieth Annual Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Cao et al., 2007] Zhe Cao, Tao Qin, Tie-Yan Liu, Ming-Feng Tsai, and Hang Li. Learning to rank: from pairwise ap- proach to listwise approach. In Machine Learning, Pro- ceedings of the Twenty-Fourth International Conference, pages 129-136, 2007.","title":"Learning to rank: from pairwise approach to listwise approach","authors":"Zhe Cao, Tao Qin, Tie-Yan Liu, Ming-Feng Tsai, Hang Li","year":"2007","venue":"Machine Learning, Proceedings of the Twenty-Fourth International Conference","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chen and Guestrin, 2016] Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowl- edge discovery and data mining, pages 785-794, 2016.","title":"Xgboost: A scalable tree boosting system","authors":"Tianqi Chen, Carlos Guestrin","year":"2016","venue":"Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Järvelin and Kekäläinen, 2017] Kalervo Järvelin and Jaana Kekäläinen. IR evaluation methods for retrieving highly relevant documents. SIGIR Forum, 51(2):243-250, 2017.","title":"IR evaluation methods for retrieving highly relevant documents","authors":"Järvelin, Kekäläinen, Kalervo Järvelin, Jaana Kekäläinen","year":"2017","venue":"SIGIR Forum","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ke et al., 2017] Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie- Yan Liu. Lightgbm: A highly efficient gradient boosting decision tree. In Advances in Neural Information Process- ing Systems 30: Annual Conference on Neural Information Processing Systems, pages 3146-3154, 2017.","title":"Lightgbm: A highly efficient gradient boosting decision tree","authors":"Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, Tie-Yan Liu","year":"2017","venue":"Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li et al., 2020] Minghan Li, Xialei Liu, Joost van de Weijer, and Bogdan C. Raducanu. Learning to rank for active learn- ing: A listwise approach. In 25th International Conference on Pattern Recognition, pages 5587-5594, 2020.","title":"Learning to rank for active learning: A listwise approach","authors":"Li","year":"2020","venue":"25th International Conference on Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2022] Yuchen Li, Haoyi Xiong, Linghe Kong, Rui Zhang, Dejing Dou, and Guihai Chen. Meta hierarchical reinforced learning to rank for recommendation: A compre- hensive study in moocs. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 302-317, 2022.","title":"Meta hierarchical reinforced learning to rank for recommendation: A comprehensive study in moocs","authors":"Yuchen Li, Haoyi Xiong, Linghe Kong, Rui Zhang, Dejing Dou, Guihai Chen","year":"2022","venue":"Joint European Conference on Machine Learning and Knowledge Discovery in Databases","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2023a] Yuchen Li, Haoyi Xiong, Linghe Kong, Zeyi Sun, Hongyang Chen, Shuaiqiang Wang, and Dawei Yin. Mpgraf: a modular and pre-trained graphformer for learning to rank at web-scale. In 2023 IEEE International Conference on Data Mining (ICDM), pages 339-348. IEEE, 2023.","title":"Mpgraf: a modular and pre-trained graphformer for learning to rank at web-scale","authors":"Li","year":"2023","venue":"IEEE International Conference on Data Mining (ICDM)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2023b] Yuchen Li, Haoyi Xiong, Linghe Kong, Qingzhong Wang, Shuaiqiang Wang, Guihai Chen, and Dawei Yin. S2phere: Semi-supervised pre-training for web search over heterogeneous learning to rank data. In Pro- ceedings of the 29th ACM SIGKDD Conference on Knowl- edge Discovery and Data Mining, pages 4437-4448, 2023.","title":"S2phere: Semi-supervised pre-training for web search over heterogeneous learning to rank data","authors":"Li","year":"2023","venue":"Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2023c] Yuchen Li, Haoyi Xiong, Linghe Kong, Shuaiqiang Wang, Zeyi Sun, Hongyang Chen, Guihai Chen, and Dawei Yin. Ltrgcn: Large-scale graph convolutional networks-based learning to rank for web search. In Joint European Conference on Machine Learning and Knowl- edge Discovery in Databases, pages 635-651. Springer, 2023.","title":"Ltrgcn: Large-scale graph convolutional networks-based learning to rank for web search","authors":"Li","year":"2023","venue":"Joint European Conference on Machine Learning and Knowledge Discovery in Databases","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2023d] Yuchen Li, Haoyi Xiong, Linghe Kong, Rui Zhang, Fanqin Xu, Guihai Chen, and Minglu Li. Mhrr: Moocs recommender service with meta hierarchical rein- forced ranking. IEEE Transactions on Services Computing, 2023.","title":"Mhrr: Moocs recommender service with meta hierarchical reinforced ranking","authors":"Li","year":"2023","venue":"IEEE Transactions on Services Computing","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li et al., 2023e] Yuchen Li, Haoyi Xiong, Qingzhong Wang, Linghe Kong, Hao Liu, Haifang Li, Jiang Bian, Shuaiqiang Wang, Guihai Chen, Dejing Dou, et al. Coltr: Semi- supervised learning to rank with co-training and over- parameterization for web search. IEEE Transactions on Knowledge and Data Engineering, 2023.","title":"Coltr: Semisupervised learning to rank with co-training and overparameterization for web search","authors":"Li","year":"2023","venue":"IEEE Transactions on Knowledge and Data Engineering","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liu et al., 2021] Yiding Liu, Weixue Lu, Suqi Cheng, Dait- ing Shi, Shuaiqiang Wang, Zhicong Cheng, and Dawei Yin. Pre-trained language model for web-scale retrieval in baidu search. In KDD '21: The 27th ACM SIGKDD Con- ference on Knowledge Discovery and Data Mining, pages 3365-3375, 2021.","title":"Pre-trained language model for web-scale retrieval in baidu search","authors":"Liu","year":"2021","venue":"KDD '21: The 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Pobrotyn and Białobrzeski, 2021] Przemysław Pobrotyn and Radosław Białobrzeski. Neuralndcg: Direct optimi- sation of a ranking metric via differentiable relaxation of sorting. arXiv preprint arXiv:2102.07831, 2021.","title":"Neuralndcg: Direct optimisation of a ranking metric via differentiable relaxation of sorting","authors":"Przemysław Pobrotyn, Radosław Białobrzeski","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Pobrotyn et al., 2020] Przemysław Pobrotyn, Tomasz Bartczak, Mikołaj Synowiec, Radosław Białobrzeski, and Jarosław Bojar. Context-aware learning to rank with self-attention. arXiv preprint arXiv:2005.10084, 2020.","title":"Context-aware learning to rank with self-attention","authors":"Przemysław Pobrotyn, Tomasz Bartczak, Mikołaj Synowiec, Radosław Białobrzeski, Jarosław Bojar","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Qin and Liu, 2013] Tao Qin and Tie-Yan Liu. Introducing letor 4.0 datasets. arXiv preprint arXiv:1306.2597, 2013.","title":"Introducing letor 4.0 datasets","authors":"Liu Qin, Tao Qin, Tie-Yan Liu","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Qin et al., 2010] Tao Qin, Tie-Yan Liu, and Hang Li. A general approximation framework for direct optimization of information retrieval measures. Inf. Retr., 13(4):375-397, 2010.","title":"A general approximation framework for direct optimization of information retrieval measures","authors":"Tao Qin, Tie-Yan Liu, Hang Li","year":"2010","venue":"Inf. Retr","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Rahimi and Recht, 2007] Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In Ad- vances in Neural Information Processing Systems 20, Pro- ceedings of the Twenty-First Annual Conference on Neural Information Processing Systems, pages 1177-1184, 2007.","title":"Random features for large-scale kernel machines","authors":"Recht Rahimi, Ali Rahimi, Benjamin Recht","year":"2007","venue":"Advances in Neural Information Processing Systems 20, Proceedings of the Twenty-First Annual Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Szummer and Yilmaz, 2011] Martin Szummer and Emine Yilmaz. Semi-supervised learning to rank with preference regularization. In Proceedings of the 20th ACM Confer- ence on Information and Knowledge Management, pages 269-278, 2011.","title":"Semi-supervised learning to rank with preference regularization","authors":"Yilmaz Szummer, Martin Szummer, Emine Yilmaz","year":"2011","venue":"Proceedings of the 20th ACM Conference on Information and Knowledge Management","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Sys- tems 30: Annual Conference on Neural Information Pro- cessing Systems, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang et al., 2021] Ruoxi Wang, Rakesh Shivanna, Derek Zhiyuan Cheng, Sagar Jain, Dong Lin, Lichan Hong, and Ed H. Chi. DCN V2: improved deep & cross network and practical lessons for web-scale learning to rank systems. In WWW '21: The Web Conference, pages 1785-1797, 2021.","title":"DCN V2: improved deep & cross network and practical lessons for web-scale learning to rank systems","authors":"Wang","year":"2021","venue":"WWW '21: The Web Conference","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Werner, 2022] Tino Werner. A review on instance ranking problems in statistical learning. Mach. Learn., 111(2):415- 463, 2022.","title":"A review on instance ranking problems in statistical learning","authors":"Tino Werner","year":"2022","venue":"Mach. Learn","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Xia et al., 2008] Fen Xia, Tie-Yan Liu, Jue Wang, Wensheng Zhang, and Hang Li. Listwise approach to learning to rank: theory and algorithm. In Machine Learning, Proceedings of the Twenty-Fifth International Conference, pages 1192- 1199, 2008.","title":"Listwise approach to learning to rank: theory and algorithm","authors":"Fen Xia, Tie-Yan Liu, Jue Wang, Wensheng Zhang, Hang Li","year":"2008","venue":"Proceedings of the Twenty-Fifth International Conference","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Yang and Ying, 2023] Tianbao Yang and Yiming Ying. AUC maximization in the era of big data and AI: A survey. ACM Comput. Surv., 55(8):172:1-172:37, 2023.","title":"AUC maximization in the era of big data and AI: A survey","authors":"Yang, Ying ; ] Tianbao Yang, Yiming Ying","year":"2023","venue":"ACM Comput. Surv","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhang et al., 2016] Xin Zhang, Ben He, and Tiejian Luo. Training query filtering for semi-supervised learning to rank with pseudo labels. World Wide Web, 19(5):833-864, 2016.","title":"Training query filtering for semi-supervised learning to rank with pseudo labels","authors":"Xin Zhang, Ben He, Tiejian Luo","year":"2016","venue":"World Wide Web","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhao et al., 2010] Shiqi Zhao, Haifeng Wang, and Ting Liu. Paraphrasing with search engine query logs. In COLING 2010, 23rd International Conference on Computational Linguistics, Proceedings of the Conference, pages 1317- 1325, 2010.","title":"Paraphrasing with search engine query logs","authors":"Shiqi Zhao, Haifeng Wang, Ting Liu","year":"2010","venue":"COLING 2010, 23rd International Conference on Computational Linguistics, Proceedings of the Conference","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhao et al., 2011] Shiqi Zhao, Haifeng Wang, Chao Li, Ting Liu, and Yi Guan. Automatically generating questions from queries for community-based question answering. In Proceedings of 5th international joint conference on natural language processing, pages 929-937, 2011.","title":"Automatically generating questions from queries for community-based question answering","authors":"Shiqi Zhao, Haifeng Wang, Chao Li, Ting Liu, Yi Guan","year":"2011","venue":"Proceedings of 5th international joint conference on natural language processing","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zou et al., 2021] Lixin Zou, Shengqiang Zhang, Hengyi Cai, Dehong Ma, Suqi Cheng, Shuaiqiang Wang, Daiting Shi, Zhicong Cheng, and Dawei Yin. Pre-trained language model based ranking in baidu search. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 4014-4022, 2021.","title":"Pre-trained language model based ranking in baidu search","authors":"Lixin Zou, Shengqiang Zhang, Hengyi Cai, Dehong Ma, Suqi Cheng, Shuaiqiang Wang, Daiting Shi, Zhicong Cheng, Dawei Yin","year":"2021","venue":"Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}