{"type":"ScholarlyArticle","title":"Unsupervised Context Aware Sentence Representation Pretraining for Multi-lingual Dense Retrieval","authors":[{"name":"Ning Wu","slug":"ning-wu","given":"Ning","family":"Wu","affiliation":"Microsoft Research Asia {wuning ,"},{"name":"Yaobo Liang","slug":"yaobo-liang","given":"Yaobo","family":"Liang","affiliation":"Microsoft Research Asia {wuning ,"},{"name":"Houxing Ren","slug":"houxing-ren","given":"Houxing","family":"Ren","affiliation":"Microsoft Research Asia {wuning ,"},{"name":"Linjun Shou","slug":"linjun-shou","given":"Linjun","family":"Shou","affiliation":"Microsoft Research Asia {wuning ,"},{"name":"Nan Duan","slug":"nan-duan","given":"Nan","family":"Duan","affiliation":"Microsoft Research Asia {wuning ,"},{"name":"Ming Gong","slug":"ming-gong","given":"Ming","family":"Gong","affiliation":"Microsoft Research Asia {wuning ,"},{"name":"Daxin Jiang","slug":"daxin-jiang","given":"Daxin","family":"Jiang","affiliation":"Microsoft Research Asia {wuning ,"}],"abstract":"Recent research demonstrates the effectiveness of using pretrained language models (PLM) to improve dense retrieval and multilingual dense retrieval. In this work, we present a simple but effective monolingual pretraining task called contrastive context prediction (CCP) to learn sentence representation by modeling sentence level contextual relation. By pushing the embedding of sentences in a local context closer and pushing random negative samples away, different languages could form isomorphic structure, then sentence pairs in two different languages will be automatically aligned.  Our experiments show that model collapse and information leakage are very easy to happen during contrastive training of language model, but language-specific memory bank and asymmetric batch normalization operation play an essential role in preventing collapsing and information leakage, respectively. Besides, a post-processing for sentence embedding is also very effective to achieve better retrieval performance.  On the multilingual sentence retrieval task Tatoeba, our model achieves new SOTA results among methods without using bilingual data. Our model also shows larger gain on Tatoeba when transferring between non-English pairs. On two multi-lingual query-passage retrieval tasks, XOR Retrieve and Mr.TYDI, our model even achieves two SOTA results in both zero-shot and supervised setting among all pretraining models using bilingual data.","doi":"10.24963/ijcai.2022/612","url":"https://ijcai.org/proceedings/2022/612","pdf":"https://ijcai.org/proceedings/2022/0612.pdf","markdown":"https://ijcai.org/proceedings/2022/612.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/612","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Natural Language Processing","track":"Main Track","number":612,"pages":{"first":4411,"last":4417},"keywords":["Natural Language Processing: Embeddings","Natural Language Processing: Information Retrieval and Text Mining","Natural Language Processing: Machine Translation and Multilinguality"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38985237","length_sec":86,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38987094","length_sec":117,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Artetxe and Schwenk, 2019] Mikel Artetxe and Holger Schwenk. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. TACL, 7:597-610, 2019.","title":"Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond","authors":"Artetxe, Schwenk, Mikel Artetxe, Holger Schwenk","year":"2019","venue":"TACL","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Asai et al., 2020] Akari Asai, Jungo Kasai, Jonathan H Clark, Kenton Lee, Eunsol Choi, and Hannaneh Hajishirzi. Xor qa: Cross-lingual open-retrieval question answering. arXiv preprint arXiv:2010.11856, 2020.","title":"Xor qa: Cross-lingual open-retrieval question answering","authors":"Akari Asai, Jungo Kasai, Jonathan H Clark, Kenton Lee, Eunsol Choi, Hannaneh Hajishirzi","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2020] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. ICML, 2020.","title":"A simple framework for contrastive learning of visual representations","authors":"Ting Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey Hinton","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chi et al., 2021] Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, and Ming Zhou. Infoxlm: An information-theoretic framework for cross-lingual language model pre-training. NAACL, 2021.","title":"Infoxlm: An information-theoretic framework for cross-lingual language model pre-training","authors":"Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, Ming Zhou","year":"2021","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Conneau and Lample, 2019] Alexis Conneau and Guillaume Lample. Cross-lingual language model pretraining. In NeuralIPS, 2019.","title":"Cross-lingual language model pretraining","authors":"Alexis Conneau, Guillaume Lample","year":"2019","venue":"NeuralIPS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Conneau et al., 2018] Alexis Conneau, Guillaume Lample, Marc'Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. Word translation without parallel data. ICLR, 2018.","title":"Word translation without parallel data","authors":"Alexis Conneau, Guillaume Lample, Marc'aurelio Ranzato, Ludovic Denoyer, Hervé Jégou","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Conneau et al., 2020] Alexis Conneau, Kartikay Khandel- wal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. ACL, 2020.","title":"Unsupervised cross-lingual representation learning at scale","authors":"Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, Veselin Stoyanov","year":"2020","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, pages 4171-4186, June 2019.","title":"BERT: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2019","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Feng et al., 2020] Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. Language- agnostic bert sentence embedding. arXiv preprint arXiv:2007.01852, 2020.","title":"Languageagnostic bert sentence embedding","authors":"Feng","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Fu et al., 2020] Zuohui Fu, Yikun Xian, Shijie Geng, Yingqiang Ge, Yuting Wang, Xin Dong, Guang Wang, and Gerard de Melo. Absent: Cross-lingual sentence represen- tation mapping with bidirectional gans. In AAAI, 2020.","title":"Absent: Cross-lingual sentence representation mapping with bidirectional gans","authors":"Fu","year":"2020","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gao and Callan, 2021] Luyu Gao and Jamie Callan. Un- supervised corpus aware language model pre-training for dense passage retrieval. arXiv preprint arXiv:2108.05540, 2021.","title":"Unsupervised corpus aware language model pre-training for dense passage retrieval","authors":"Callan Gao, Luyu Gao, Jamie Callan","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Guu et al., 2020] Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Realm: Retrieval- augmented language model pre-training. arXiv preprint arXiv:2002.08909, 2020.","title":"Realm: Retrievalaugmented language model pre-training","authors":"Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, Ming-Wei Chang","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[He et al., 2020] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsu- pervised visual representation learning. In CVPR, pages 9729-9738, 2020.","title":"Momentum contrast for unsupervised visual representation learning","authors":"Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, Ross Girshick","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Huang et al., 2019] Haoyang Huang, Yaobo Liang, Nan Duan, Ming Gong, Linjun Shou, Daxin Jiang, and Ming Zhou. Unicoder: A universal language encoder by pre- training with multiple cross-lingual tasks. In EMNLP, 2019.","title":"Unicoder: A universal language encoder by pretraining with multiple cross-lingual tasks","authors":"Haoyang Huang, Yaobo Liang, Nan Duan, Ming Gong, Linjun Shou, Daxin Jiang, Ming Zhou","year":"2019","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Karpukhin et al., 2020] Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04906, 2020.","title":"Dense passage retrieval for open-domain question answering","authors":"Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, Wen-Tau Yih","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lee et al., 2019] Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. Latent retrieval for weakly supervised open domain question answering. In Proceedings of ACL, pages 6086-6096, 2019.","title":"Latent retrieval for weakly supervised open domain question answering","authors":"Lee","year":"2019","venue":"Proceedings of ACL","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liang et al., 2020] Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, Xiaodong Fan, Ruofei Zhang, Rahul Agrawal, Edward Cui, Sining Wei, Taroon Bharti, Ying Qiao, Jiun-Hung Chen, Winnie Wu, Shuguang Liu, Fan Yang, Daniel Campos, Rangan Majumder, and Ming Zhou. XGLUE: A new benchmark datasetfor cross- lingual pre-training, understanding and generation. In EMNLP, pages 6008-6018, 2020.","title":"XGLUE: A new benchmark datasetfor crosslingual pre-training, understanding and generation","authors":"Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, Xiaodong Fan, Ruofei Zhang, Rahul Agrawal, Edward Cui, Sining Wei, Taroon Bharti, Ying Qiao, Jiun-Hung Chen, Winnie Wu, Shuguang Liu, Fan Yang, Daniel Campos, Rangan Majumder, Ming Zhou","year":"2020","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Liu et al., 2019] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019.","title":"Roberta: A robustly optimized bert pretraining approach","authors":"Liu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Oord et al., 2018] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018.","title":"Representation learning with contrastive predictive coding","authors":"Aaron Van Den Oord, Yazhe Li, Oriol Vinyals","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Qu et al., 2021] Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. RocketQA: An optimized training ap- proach to dense passage retrieval for open-domain question answering. In Proceedings of NAACL-HLT, 2021.","title":"RocketQA: An optimized training approach to dense passage retrieval for open-domain question answering","authors":"Qu","year":"2021","venue":"Proceedings of NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Tran et al., 2020] Chau Tran, Yuqing Tang, Xian Li, and Ji- atao Gu. Cross-lingual retrieval for iterative self-supervised training. arXiv preprint arXiv:2006.09526, 2020.","title":"Cross-lingual retrieval for iterative self-supervised training","authors":"Chau Tran, Yuqing Tang, Xian Li, Jiatao Gu","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wenzek et al., 2019] Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzman, Armand Joulin, and Edouard Grave. Ccnet: Extracting high quality monolingual datasets from web crawl data. arXiv preprint arXiv:1911.00359, 2019.","title":"Ccnet: Extracting high quality monolingual datasets from web crawl data","authors":"Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzman, Armand Joulin, Edouard Grave","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wu et al., 2018] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In CVPR, pages 3733- 3742, 2018.","title":"Unsupervised feature learning via nonparametric instance discrimination","authors":"Zhirong Wu, Yuanjun Xiong, Stella X Yu, Dahua Lin","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wu et al., 2020] Zhuofeng Wu, Sinong Wang, Jiatao Gu, Madian Khabsa, Fei Sun, and Hao Ma. Clear: Con- trastive learning for sentence representation. arXiv preprint arXiv:2012.15466, 2020.","title":"Clear: Contrastive learning for sentence representation","authors":"Wu","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Xiong et al., 2020] Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. Approximate nearest neighbor neg- ative contrastive learning for dense text retrieval. arXiv preprint arXiv:2007.00808, 2020.","title":"Approximate nearest neighbor negative contrastive learning for dense text retrieval","authors":"Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, Arnold Overwijk","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhang et al., 2021] Xinyu Zhang, Xueguang Ma, Peng Shi, and Jimmy Lin. Mr. tydi: A multi-lingual benchmark for dense retrieval. arXiv preprint arXiv:2108.08787, 2021.","title":"Mr. tydi: A multi-lingual benchmark for dense retrieval","authors":"Xinyu Zhang, Xueguang Ma, Peng Shi, Jimmy Lin","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}