{"type":"ScholarlyArticle","title":"MPGraf: a Modular and Pre-trained Graphformer for Learning to Rank at Web-scale (Extended Abstract)","authors":[{"name":"Yuchen Li","slug":"yuchen-li","given":"Yuchen","family":"Li","affiliation":"Baidu Inc. , China; Shanghai Jiao Tong University , China"},{"name":"Haoyi Xiong","slug":"haoyi-xiong","given":"Haoyi","family":"Xiong","affiliation":"Baidu Inc. , China"},{"name":"Linghe Kong","slug":"linghe-kong","given":"Linghe","family":"Kong","affiliation":"Shanghai Jiao Tong University , China"},{"name":"Zeyi Sun","slug":"zeyi-sun","given":"Zeyi","family":"Sun","affiliation":"Zhejiang Lab , China"},{"name":"Hongyang Chen","slug":"hongyang-chen","given":"Hongyang","family":"Chen","affiliation":"Zhejiang Lab , China"},{"name":"Shuaiqiang Wang","slug":"shuaiqiang-wang","given":"Shuaiqiang","family":"Wang","affiliation":"Baidu Inc. , China"},{"name":"Dawei Yin","slug":"dawei-yin","given":"Dawei","family":"Yin","affiliation":"Baidu Inc. , China"}],"abstract":"Both Transformer and Graph Neural Networks (GNNs) have been used in learning to rank (LTR), however, they adhere to two distinct yet complementary problem formulations, i.e., ranking score regression based on query-webpage pairs and link prediction within query-webpage bipartite graphs, respectively. Though it is possible to pre-train GNNs or Transformers on source datasets and fine-tune them subject to sparsely annotated LTR datasets separately, the source-target distribution shifts across the pairs and bipartite graphs domains make it extremely difficult to integrate these diverse models into a single LTR framework at a web-scale. We introduce the novel MPGraf model, which utilizes a modular and capsule-based pre-training approach, aiming to incorporate regression capacities from Transformers and link prediction capabilities of GNNs cohesively. \r\nWe conduct extensive experiments to evaluate the performance of MPGraf using real-world datasets collected from large-scale search engines. The results show that MPGraf can outperform baseline algorithms on several major metrics. Further, we deploy and evaluate MPGraf atop a large-scale search engine with realistic web traffic via A/B tests, where we can still observe significant improvement. MPGraf performs consistently in both offline and online evaluations.","doi":"10.24963/ijcai.2024/937","url":"https://ijcai.org/proceedings/2024/937","pdf":"https://ijcai.org/proceedings/2024/0937.pdf","markdown":"https://ijcai.org/proceedings/2024/937.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/937","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Sister Conferences Best Papers","track":"Sister Conferences Best Papers","number":937,"pages":{"first":8439,"last":8443},"keywords":["Data Mining: DM: Mining text, web, social media"],"videos":[],"references":[{"position":1,"raw":"[Burges et al., 2006] Christopher J. C. Burges, Robert Ragno, and Quoc Viet Le. Learning to rank with nonsmooth cost functions. In Advances in Neural Information Processing Systems 19, Proceedings of the Twentieth Annual Confer- ence on Neural Information Processing Systems, pages 193-200, 2006.","title":"Learning to rank with nonsmooth cost functions","authors":"J C Christopher, Robert Burges, Quoc Ragno, Le Viet","year":"2006","venue":"Proceedings of the Twentieth Annual Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cao et al., 2007] Zhe Cao, Tao Qin, Tie-Yan Liu, Ming-Feng Tsai, and Hang Li. Learning to rank: from pairwise ap- proach to listwise approach. In Proceedings of the 24th international conference on Machine learning, pages 129- 136, 2007.","title":"Learning to rank: from pairwise approach to listwise approach","authors":"Zhe Cao, Tao Qin, Tie-Yan Liu, Ming-Feng Tsai, Hang Li","year":"2007","venue":"Proceedings of the 24th international conference on Machine learning","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen and Guestrin, 2016] Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowl- edge discovery and data mining, pages 785-794, 2016.","title":"Xgboost: A scalable tree boosting system","authors":"Tianqi Chen, Carlos Guestrin","year":"2016","venue":"Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chuklin et al., 2015] Aleksandr Chuklin, Anne Schuth, Ke Zhou, and Maarten De Rijke. A comparative analy- sis of interleaving methods for aggregated search. ACM Transactions on Information Systems (TOIS), 33(2):1-38, 2015.","title":"A comparative analysis of interleaving methods for aggregated search","authors":"Aleksandr Chuklin, Anne Schuth, Ke Zhou, Maarten De, Rijke","year":"2015","venue":"ACM Transactions on Information Systems (TOIS)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[He et al., 2020] Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, Yong-Dong Zhang, and Meng Wang. Lightgcn: Sim- plifying and powering graph convolution network for rec- ommendation. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR, pages 639-648, 2020.","title":"Lightgcn: Simplifying and powering graph convolution network for recommendation","authors":"He","year":"2020","venue":"Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Ke et al., 2017] Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie- Yan Liu. Lightgbm: A highly efficient gradient boosting decision tree. In Advances in Neural Information Process- ing Systems 30: Annual Conference on Neural Information Processing Systems, pages 3146-3154, 2017.","title":"Lightgbm: A highly efficient gradient boosting decision tree","authors":"Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, Tie-Yan Liu","year":"2017","venue":"Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Kirichenko et al., 2022] Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Last layer re-training is suffi- cient for robustness to spurious correlations. arXiv preprint arXiv:2204.02937, 2022.","title":"Last layer re-training is sufficient for robustness to spurious correlations","authors":"Polina Kirichenko, Pavel Izmailov, Andrew Gordon, Wilson","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Li et al., 2022] Yuchen Li, Haoyi Xiong, Linghe Kong, Rui Zhang, Dejing Dou, and Guihai Chen. Meta hierarchical reinforced learning to rank for recommendation: A compre- hensive study in moocs. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 302-317, 2022.","title":"Meta hierarchical reinforced learning to rank for recommendation: A comprehensive study in moocs","authors":"Yuchen Li, Haoyi Xiong, Linghe Kong, Rui Zhang, Dejing Dou, Guihai Chen","year":"2022","venue":"Joint European Conference on Machine Learning and Knowledge Discovery in Databases","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Li et al., 2023a] Yuchen Li, Haoyi Xiong, Linghe Kong, Qingzhong Wang, Shuaiqiang Wang, Guihai Chen, and Dawei Yin. S2phere: Semi-supervised pre-training for web search over heterogeneous learning to rank data. In Pro- ceedings of the 29th ACM SIGKDD Conference on Knowl- edge Discovery and Data Mining, pages 4437-4448, 2023.","title":"S2phere: Semi-supervised pre-training for web search over heterogeneous learning to rank data","authors":"Li","year":"2023","venue":"Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li et al., 2023b] Yuchen Li, Haoyi Xiong, Linghe Kong, Shuaiqiang Wang, Zeyi Sun, Hongyang Chen, Guihai Chen, and Dawei Yin. Ltrgcn: Large-scale graph convolutional networks-based learning to rank for web search. In Joint European Conference on Machine Learning and Knowl- edge Discovery in Databases, pages 635-651. Springer, 2023.","title":"Ltrgcn: Large-scale graph convolutional networks-based learning to rank for web search","authors":"Li","year":"2023","venue":"Joint European Conference on Machine Learning and Knowledge Discovery in Databases","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2023c] Yuchen Li, Haoyi Xiong, Linghe Kong, Rui Zhang, Fanqin Xu, Guihai Chen, and Minglu Li. Mhrr: Moocs recommender service with meta hierarchical rein- forced ranking. IEEE Transactions on Services Computing, 2023.","title":"Mhrr: Moocs recommender service with meta hierarchical reinforced ranking","authors":"Li","year":"2023","venue":"IEEE Transactions on Services Computing","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2023d] Yuchen Li, Haoyi Xiong, Qingzhong Wang, Linghe Kong, Hao Liu, Haifang Li, Jiang Bian, Shuaiqiang Wang, Guihai Chen, Dejing Dou, et al. Coltr: Semi- supervised learning to rank with co-training and over- parameterization for web search. IEEE Transactions on Knowledge and Data Engineering, 2023.","title":"Coltr: Semisupervised learning to rank with co-training and overparameterization for web search","authors":"Li","year":"2023","venue":"IEEE Transactions on Knowledge and Data Engineering","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2024] Yuchen Li, Haoyi Xiong, Linghe Kong, Jiang Bian, Shuaiqiang Wang, Guihai Chen, and Dawei Yin. Gs2p: a generative pre-trained learning to rank model with over-parameterization for web-scale search. Machine Learning, pages 1-19, 2024.","title":"Gs2p: a generative pre-trained learning to rank model with over-parameterization for web-scale search","authors":"Yuchen Li, Haoyi Xiong, Linghe Kong, Jiang Bian, Shuaiqiang Wang, Guihai Chen, Dawei Yin","year":"2024","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Pobrotyn and Białobrzeski, 2021] Przemysław Pobrotyn and Radosław Białobrzeski. Neuralndcg: Direct optimi- sation of a ranking metric via differentiable relaxation of sorting. arXiv preprint arXiv:2102.07831, 2021.","title":"Neuralndcg: Direct optimisation of a ranking metric via differentiable relaxation of sorting","authors":"Przemysław Pobrotyn, Radosław Białobrzeski","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Pobrotyn et al., 2020] Przemysław Pobrotyn, Tomasz Bartczak, Mikołaj Synowiec, Radosław Białobrzeski, and Jarosław Bojar. Context-aware learning to rank with self-attention. arXiv preprint arXiv:2005.10084, 2020.","title":"Context-aware learning to rank with self-attention","authors":"Przemysław Pobrotyn, Tomasz Bartczak, Mikołaj Synowiec, Radosław Białobrzeski, Jarosław Bojar","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Qiang et al., 2023] Jipeng Qiang, Feng Zhang, Yun Li, Yun- hao Yuan, Yi Zhu, and Xindong Wu. Unsupervised statisti- cal text simplification using pre-trained language modeling for initialization. Frontiers Comput. Sci., 17(1):171303, 2023.","title":"Unsupervised statistical text simplification using pre-trained language modeling for initialization","authors":"Jipeng Qiang, Feng Zhang, Yun Li, Yunhao Yuan, Yi Zhu, Xindong Wu","year":"2023","venue":"Frontiers Comput. Sci","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Qin and Liu, 2013] Tao Qin and Tie-Yan Liu. Introducing letor 4.0 datasets. arXiv preprint arXiv:1306.2597, 2013.","title":"Introducing letor 4.0 datasets","authors":"Liu Qin, Tao Qin, Tie-Yan Liu","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Sys- tems 30: Annual Conference on Neural Information Pro- cessing Systems, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Yang et al., 2021] Junhan Yang, Zheng Liu, Shitao Xiao, Chaozhuo Li, Defu Lian, Sanjay Agrawal, Amit Singh, Guangzhong Sun, and Xing Xie. Graphformers: Gnn- nested transformers for representation learning on textual graph. Advances in Neural Information Processing Systems, 34:28798-28810, 2021.","title":"Graphformers: Gnnnested transformers for representation learning on textual graph","authors":"Junhan Yang, Zheng Liu, Shitao Xiao, Chaozhuo Li, Defu Lian, Sanjay Agrawal, Amit Singh, Guangzhong Sun, Xing Xie","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Zhao et al., 2011] Shiqi Zhao, Haifeng Wang, Chao Li, Ting Liu, and Yi Guan. Automatically generating questions from queries for community-based question answering. In Proceedings of 5th international joint conference on natural language processing, pages 929-937, 2011.","title":"Automatically generating questions from queries for community-based question answering","authors":"Shiqi Zhao, Haifeng Wang, Chao Li, Ting Liu, Yi Guan","year":"2011","venue":"Proceedings of 5th international joint conference on natural language processing","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}