{"type":"ScholarlyArticle","title":"Towards Robust Dense Retrieval via Local Ranking Alignment","authors":[{"name":"Xuanang Chen","slug":"xuanang-chen","given":"Xuanang","family":"Chen","affiliation":"University of Chinese Academy of Sciences , Beijing , China; Institute of Software , Chinese Academy of Sciences , Beijing , China"},{"name":"Jian Luo","slug":"jian-luo","given":"Jian","family":"Luo","affiliation":"University of Chinese Academy of Sciences , Beijing , China; Institute of Software , Chinese Academy of Sciences , Beijing , China"},{"name":"Ben He","slug":"ben-he","given":"Ben","family":"He","affiliation":"University of Chinese Academy of Sciences , Beijing , China; Institute of Software , Chinese Academy of Sciences , Beijing , China"},{"name":"Le Sun","slug":"le-sun","given":"Le","family":"Sun","affiliation":"Institute of Software , Chinese Academy of Sciences , Beijing , China"},{"name":"Yingfei Sun","slug":"yingfei-sun","given":"Yingfei","family":"Sun","affiliation":"University of Chinese Academy of Sciences , Beijing , China"}],"abstract":"Dense retrieval (DR) has extended the employment of pre-trained language models, like BERT, for text ranking. However, recent studies have raised the robustness issue of DR model against query variations, like query with typos, along with non-trivial performance losses. Herein, we argue that it would be beneficial to allow the DR model to learn to align the relative positions of query-passage pairs in the representation space, as query variations cause the query vector to drift away from its original position, affecting the subsequent DR effectiveness. To this end, we propose RoDR, a novel robust DR model that learns to calibrate the in-batch local ranking of query variation to that of original query for the DR space alignment. Extensive experiments on MS MARCO and ANTIQUE datasets show that RoDR significantly improves the retrieval results on both the original queries and different types of query variations. Meanwhile, RoDR provides a general query noise-tolerate learning framework that boosts the robustness and effectiveness of various existing DR models. Our code and models are openly available at https://github.com/cxa-unique/RoDR.","doi":"10.24963/ijcai.2022/275","url":"https://ijcai.org/proceedings/2022/275","pdf":"https://ijcai.org/proceedings/2022/0275.pdf","markdown":"https://ijcai.org/proceedings/2022/275.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/275","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Data Mining","track":"Main Track","number":275,"pages":{"first":1980,"last":1986},"keywords":["Machine Learning: Representation learning","Data Mining: Information Retrieval","Natural Language Processing: Information Retrieval and Text Mining","Machine Learning: Robustness"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986729","length_sec":85,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984872","length_sec":343,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Bailey et al., 2015] Peter Bailey, Alistair Moffat, Falk Sc- holer, and Paul Thomas. User variability and IR system evaluation. In SIGIR, pages 625-634, 2015.","title":"User variability and IR system evaluation","authors":"Bailey","year":"2015","venue":"SIGIR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understand- ing. In NAACL-HLT, pages 4171-4186, 2019.","title":"BERT: pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2019","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Gao et al., 2022] Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. Tevatron: An efficient and flexible toolkit for dense retrieval. CoRR, abs/2203.05765, 2022.","title":"Tevatron: An efficient and flexible toolkit for dense retrieval","authors":"Gao","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Gui et al., 2021] Tao Gui, Xiao Wang, Qi Zhang, Qin Liu, Yicheng Zou, Xin Zhou, Rui Zheng, Chong Zhang, Qinzhuo Wu, Jiacheng Ye, Zexiong Pang, Yongxin Zhang, Zhengyan Li, Ruotian Ma, Zichu Fei, Ruijian Cai, Jun Zhao, Xinwu Hu, Zhiheng Yan, Yiding Tan, Yuan Hu, Qiyuan Bian, Zhihua Liu, Bolin Zhu, Shan Qin, Xiaoyu Xing, Jinlan Fu, Yue Zhang, Minlong Peng, Xiaoqing Zheng, Yaqian Zhou, Zhongyu Wei, Xipeng Qiu, and Xu- anjing Huang. TextFlint: Unified multilingual robustness evaluation toolkit for natural language processing. CoRR, abs/2103.11441, 2021.","title":"TextFlint: Unified multilingual robustness evaluation toolkit for natural language processing","authors":"Tao Gui, Xiao Wang, Qi Zhang, Qin Liu, Yicheng Zou, Xin Zhou, Rui Zheng, Chong Zhang, Qinzhuo Wu, Jiacheng Ye, Zexiong Pang, Yongxin Zhang, Zhengyan Li, Ruotian Ma, Zichu Fei, Ruijian Cai, Jun Zhao, Xinwu Hu, Zhiheng Yan, Yiding Tan, Yuan Hu, Qiyuan Bian, Zhihua Liu, Bolin Zhu, Shan Qin, Xiaoyu Xing, Jinlan Fu, Yue Zhang, Minlong Peng, Xiaoqing Zheng, Yaqian Zhou, Zhongyu Wei, Xipeng Qiu, Xuanjing Huang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Hashemi et al., 2020] Helia Hashemi, Mohammad Alianne- jadi, Hamed Zamani, and W. Bruce Croft. ANTIQUE: A non-factoid question answering benchmark. In ECIR, pages 166-173, 2020.","title":"ANTIQUE: A non-factoid question answering benchmark","authors":"Helia Hashemi, Mohammad Aliannejadi, Hamed Zamani, W Bruce Croft","year":"2020","venue":"ECIR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hofstätter et al., 2021] Sebastian Hofstätter, Sheng-Chieh Lin, Jheng-Hong Yang, Jimmy Lin, and Allan Hanbury. Efficiently teaching an effective dense retriever with bal- anced topic aware sampling. In SIGIR, pages 113-122, 2021.","title":"Efficiently teaching an effective dense retriever with balanced topic aware sampling","authors":"Sebastian Hofstätter, Sheng-Chieh Lin, Jheng-Hong Yang, Jimmy Lin, Allan Hanbury","year":"2021","venue":"SIGIR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Karpukhin et al., 2020] Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick S. H. Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage re- trieval for open-domain question answering. In EMNLP, pages 6769-6781, 2020.","title":"Dense passage retrieval for open-domain question answering","authors":"Vladimir Karpukhin, Barlas Oguz, Sewon Min, S H Patrick, Ledell Lewis, Sergey Wu, Danqi Edunov, Wen-Tau Chen, Yih","year":"2020","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Li et al., 2021] Yizhi Li, Zhenghao Liu, Chenyan Xiong, and Zhiyuan Liu. More robust dense retrieval with con- trastive dual learning. In ICTIR, pages 287-296, 2021.","title":"More robust dense retrieval with contrastive dual learning","authors":"Yizhi Li, Zhenghao Liu, Chenyan Xiong, Zhiyuan Liu","year":"2021","venue":"ICTIR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Luan et al., 2021] Yi Luan, Jacob Eisenstein, Kristina Toutanova, and Michael Collins. Sparse, dense, and at- tentional representations for text retrieval. TACL, pages 329-345, 2021.","title":"Sparse, dense, and attentional representations for text retrieval","authors":"Yi Luan, Jacob Eisenstein, Kristina Toutanova, Michael Collins","year":"2021","venue":"TACL","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ma et al., 2021] Xiaofei Ma, Cícero Nogueira dos Santos, and Andrew O. Arnold. Contrastive fine-tuning improves robustness for neural rankers. In ACL/IJCNLP (Findings), pages 570-582, 2021.","title":"Contrastive fine-tuning improves robustness for neural rankers","authors":"Xiaofei Ma, Cícero Nogueira Dos Santos, Andrew O Arnold","year":"2021","venue":"ACL/IJCNLP (Findings)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[MacAvaney et al., 2021] Sean MacAvaney, Andrew Yates, Sergey Feldman, Doug Downey, Arman Cohan, and Nazli Goharian. Simplified data wrangling with ir datasets. In SIGIR, pages 2429-2436, 2021.","title":"Simplified data wrangling with ir datasets","authors":"Sean Macavaney, Andrew Yates, Sergey Feldman, Doug Downey, Arman Cohan, Nazli Goharian","year":"2021","venue":"SIGIR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Morris et al., 2020] John X. Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. TextAttack: A framework for adversarial attacks, data augmentation, and adversarial training in NLP. In EMNLP (Demos), pages 119-126, 2020.","title":"TextAttack: A framework for adversarial attacks, data augmentation, and adversarial training in NLP","authors":"John X Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, Yanjun Qi","year":"2020","venue":"EMNLP (Demos)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Nguyen et al., 2016] Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. MS MARCO: A human generated machine reading comprehension dataset. In CoCo@NIPS, CEUR Workshop Proceedings, 2016.","title":"MS MARCO: A human generated machine reading comprehension dataset","authors":"Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, Li Deng","year":"2016","venue":"CoCo@NIPS, CEUR Workshop Proceedings","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Penha et al., 2022] Gustavo Penha, Arthur Câmara, and Claudia Hauff. Evaluating the robustness of retrieval pipelines with query variation generators. In ECIR, pages 397-412, 2022.","title":"Evaluating the robustness of retrieval pipelines with query variation generators","authors":"Gustavo Penha, Arthur Câmara, Claudia Hauff","year":"2022","venue":"ECIR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Qu et al., 2021] Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. RocketQA: An optimized training ap- proach to dense passage retrieval for open-domain ques- tion answering. In NAACL-HLT, pages 5835-5847, 2021.","title":"RocketQA: An optimized training approach to dense passage retrieval for open-domain question answering","authors":"Qu","year":"2021","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Reimers and Gurevych, 2019] Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using siamese BERT-networks. In EMNLP/IJCNLP, pages 3980-3990, 2019.","title":"Sentence-BERT: Sentence embeddings using siamese BERT-networks","authors":"Gurevych Reimers, Nils Reimers, Iryna Gurevych","year":"2019","venue":"EMNLP/IJCNLP","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Ren et al., 2021a] Ruiyang Ren, Shangwen Lv, Yingqi Qu, Jing Liu, Wayne Xin Zhao, Qiaoqiao She, Hua Wu, Haifeng Wang, and Ji-Rong Wen. PAIR: leveraging passage-centric similarity relation for improving dense passage retrieval. In ACL/IJCNLP (Findings), pages 2173-2183, 2021.","title":"PAIR: leveraging passage-centric similarity relation for improving dense passage retrieval","authors":"Ren","year":"2021","venue":"ACL/IJCNLP (Findings)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Ren et al., 2021b] Ruiyang Ren, Yingqi Qu, Jing Liu, Wayne Xin Zhao, Qiaoqiao She, Hua Wu, Haifeng Wang, and Ji-Rong Wen. RocketQAv2: A joint training method for dense passage retrieval and passage re-ranking. In EMNLP, pages 2825-2835, 2021.","title":"RocketQAv2: A joint training method for dense passage retrieval and passage re-ranking","authors":"Ren","year":"2021","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wu et al., 2021] Chen Wu, Ruqing Zhang, Jiafeng Guo, Yixing Fan, and Xueqi Cheng. Are neural ranking models robust? CoRR, abs/2108.05018, 2021.","title":"Are neural ranking models robust?","authors":"Chen Wu, Ruqing Zhang, Jiafeng Guo, Yixing Fan, Xueqi Cheng","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Xiong et al., 2021] Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. Approximate nearest neighbor negative contrastive learning for dense text re- trieval. In ICLR, 2021.","title":"Approximate nearest neighbor negative contrastive learning for dense text retrieval","authors":"Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N Bennett, Junaid Ahmed, Arnold Overwijk","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Zhan et al., 2020] Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Min Zhang, and Shaoping Ma. RepBERT: Contextu- alized text embeddings for first-stage retrieval. CoRR, abs/2006.15498, 2020.","title":"RepBERT: Contextualized text embeddings for first-stage retrieval","authors":"Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Min Zhang, Shaoping Ma","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zhan et al., 2021] Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Ji- afeng Guo, Min Zhang, and Shaoping Ma. Optimizing dense retrieval model training with hard negatives. In SI- GIR, pages 1503-1512, 2021.","title":"Optimizing dense retrieval model training with hard negatives","authors":"Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Jiafeng Guo, Min Zhang, Shaoping Ma","year":"2021","venue":"SI-GIR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhuang and Zuccon, 2021] Shengyao Zhuang and Guido Zuccon. Dealing with typos for bert-based passage re- trieval and ranking. In EMNLP, pages 2836-2842, 2021.","title":"Dealing with typos for bert-based passage retrieval and ranking","authors":"Zuccon Zhuang, Shengyao Zhuang, Guido Zuccon","year":"2021","venue":"EMNLP","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}