{"type":"ScholarlyArticle","title":"Learning Discrete Representations via Constrained Clustering for Effective and Efficient Dense Retrieval (Extended Abstract)","authors":[{"name":"Jingtao Zhan","slug":"jingtao-zhan","given":"Jingtao","family":"Zhan","affiliation":"Department of Computer Science and Technology , Beijing National Research Center for Information Science and Technology , Tsinghua University , Beijing 100084 , China"},{"name":"Jiaxin Mao","slug":"jiaxin-mao","given":"Jiaxin","family":"Mao","affiliation":"Beijing Key Laboratory of Big Data Management and Analysis Methods , Gaoling School of Artificial Intelligence , Renmin University of China , Beijing 100872 , China"},{"name":"Yiqun Liu","slug":"yiqun-liu","given":"Yiqun","family":"Liu","affiliation":"Department of Computer Science and Technology , Beijing National Research Center for Information Science and Technology , Tsinghua University , Beijing 100084 , China"},{"name":"Jiafeng Guo","slug":"jiafeng-guo","given":"Jiafeng","family":"Guo","affiliation":"CAS Key Lab of Network Data Science and Technology , Institute of Computing Technology , Chinese Academy of Sciences , Beijing , China"},{"name":"Min Zhang","slug":"min-zhang","given":"Min","family":"Zhang","affiliation":"Department of Computer Science and Technology , Beijing National Research Center for Information Science and Technology , Tsinghua University , Beijing 100084 , China"},{"name":"Shaoping Ma","slug":"shaoping-ma","given":"Shaoping","family":"Ma","affiliation":"Department of Computer Science and Technology , Beijing National Research Center for Information Science and Technology , Tsinghua University , Beijing 100084 , China"}],"abstract":"Dense Retrieval~(DR) has achieved state-of-the-art first-stage ranking effectiveness. However, the efficiency of most existing DR models is limited by the large memory cost of storing dense vectors and the time-consuming nearest neighbor search~(NNS) in vector space. Therefore, we present RepCONC, a novel retrieval model that learns discrete Representations via CONstrained Clustering. RepCONC jointly trains dual-encoders and the Product Quantization~(PQ) method to learn discrete document representations and enables fast approximate NNS with compact indexes. It models quantization as a constrained clustering process, which requires the document embeddings to be uniformly clustered around the quantization centroids. We theoretically demonstrate that the uniform clustering constraint facilitates representation distinguishability. Extensive experiments show that RepCONC substantially outperforms a wide range of existing retrieval models in terms of retrieval effectiveness, memory efficiency, and time efficiency.","doi":"10.24963/ijcai.2023/728","url":"https://ijcai.org/proceedings/2023/728","pdf":"https://ijcai.org/proceedings/2023/0728.pdf","markdown":"https://ijcai.org/proceedings/2023/728.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/728","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Sister Conferences Best Papers","track":"Sister Conferences Best Papers","number":728,"pages":{"first":6504,"last":6508},"keywords":["Sister Conferences Best Papers: Search","Sister Conferences Best Papers: Machine Learning","Sister Conferences Best Papers: Natural Language Processing","Sister Conferences Best Papers: Data Mining"],"videos":[],"references":[{"position":1,"raw":"[Bajaj et al., 2016] Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Ran- gan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. Ms marco: A human generated ma- chine reading comprehension dataset. arXiv preprint arXiv:1611.09268, 2016.","title":"Ms marco: A human generated machine reading comprehension dataset","authors":"Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew Mcnamara, Bhaskar Mitra, Tri Nguyen","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2020] Ting Chen, Lala Li, and Yizhou Sun. Dif- ferentiable product quantization for end-to-end embedding compression. In International Conference on Machine Learning, pages 1617-1626. PMLR, 2020.","title":"Differentiable product quantization for end-to-end embedding compression","authors":"Ting Chen, Lala Li, Yizhou Sun","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Craswell et al., 2020] Nick Craswell, Bhaskar Mitra, Em- ine Yilmaz, Daniel Campos, and Ellen M. Voorhees. Overview of the trec 2019 deep learning track. In Text REtrieval Conference (TREC). TREC, 2020.","title":"Overview of the trec 2019 deep learning track","authors":"Nick Craswell, Mitra Bhaskar, Emine Yilmaz, Daniel Campos, Ellen M Voorhees","year":"2020","venue":"Text REtrieval Conference (TREC). TREC","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cuturi, 2013] Marco Cuturi. Sinkhorn distances: Light- speed computation of optimal transport. Advances in neu- ral information processing systems, 26:2292-2300, 2013.","title":"Sinkhorn distances: Lightspeed computation of optimal transport","authors":"Marco Cuturi","year":"2013","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dai and Callan, 2019] Zhuyun Dai and Jamie Callan. Context-aware sentence/passage term importance es- timation for first stage retrieval. arXiv preprint arXiv:1910.10687, 2019.","title":"Context-aware sentence/passage term importance estimation for first stage retrieval","authors":"Callan Dai, Zhuyun Dai, Jamie Callan","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gao et al., 2021] Luyu Gao, Zhuyun Dai, and Jamie Callan. COIL: Revisit exact lexical match in information retrieval with contextualized inverted list. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, pages 3030-3042, Online, June 2021.","title":"COIL: Revisit exact lexical match in information retrieval with contextualized inverted list","authors":"Gao","year":"2021","venue":"Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Ge et al., 2013] Tiezheng Ge, Kaiming He, Qifa Ke, and Jian Sun. Optimized product quantization. IEEE trans- actions on pattern analysis and machine intelligence, 36(4):744-755, 2013.","title":"Optimized product quantization","authors":"Ge","year":"2013","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gong et al., 2012] Yunchao Gong, Svetlana Lazebnik, Al- bert Gordo, and Florent Perronnin. Iterative quantization: A procrustean approach to learning binary codes for large- scale image retrieval. IEEE transactions on pattern anal- ysis and machine intelligence, 35(12):2916-2929, 2012.","title":"Iterative quantization: A procrustean approach to learning binary codes for largescale image retrieval","authors":"Gong","year":"2012","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Guo et al., 2020] Ruiqi Guo, Philip Sun, Erik Lindgren, Quan Geng, David Simcha, Felix Chern, and Sanjiv Ku- mar. Accelerating large-scale inference with anisotropic vector quantization. In International Conference on Ma- chine Learning, pages 3887-3896. PMLR, 2020.","title":"Accelerating large-scale inference with anisotropic vector quantization","authors":"Ruiqi Guo, Philip Sun, Erik Lindgren, Quan Geng, David Simcha, Felix Chern, Sanjiv Kumar","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jegou et al., 2010] Herve Jegou, Matthijs Douze, and Cordelia Schmid. Product quantization for nearest neigh- bor search. IEEE transactions on pattern analysis and ma- chine intelligence, 33(1):117-128, 2010.","title":"Product quantization for nearest neighbor search","authors":"Jegou","year":"2010","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Karpukhin et al., 2020] Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), 2020.","title":"Dense passage retrieval for open-domain question answering","authors":"Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, Wen-Tau Yih","year":"2020","venue":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Khattab and Zaharia, 2020] O. Khattab and M. Zaharia. Colbert: Efficient and effective passage search via con- textualized late interaction over bert. Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, 2020.","title":"Colbert: Efficient and effective passage search via contextualized late interaction over bert","authors":"O Khattab, M Zaharia","year":"2020","venue":"Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lin and Ma, 2021] Jimmy Lin and Xueguang Ma. A few brief notes on deepimpact, coil, and a conceptual frame- work for information retrieval techniques. arXiv preprint arXiv:2106.14807, 2021.","title":"Jimmy Lin and Xueguang Ma. A few brief notes on deepimpact, coil, and a conceptual framework for information retrieval techniques","authors":"Ma Lin","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lin et al., 2020] Sheng-Chieh Lin, Jheng-Hong Yang, and Jimmy Lin. Distilling dense representations for rank- ing using tightly-coupled teachers. arXiv preprint arXiv:2010.11386, 2020.","title":"Distilling dense representations for ranking using tightly-coupled teachers","authors":"Lin","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mallia et al., 2021] Antonio Mallia, Omar Khattab, Torsten Suel, and Nicola Tonellotto. Learning passage impacts for inverted indexes. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '21, 2021.","title":"Learning passage impacts for inverted indexes","authors":"Mallia","year":"2021","venue":"Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '21","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Nogueira et al., 2019a] Rodrigo Nogueira, Jimmy Lin, and AI Epistemic. From doc2query to doctttttquery. Online preprint, 2019.","title":"From doc2query to doctttttquery","authors":"Nogueira","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Nogueira et al., 2019b] Rodrigo Nogueira, Wei Yang, Jimmy Lin, and Kyunghyun Cho. Document expansion by query prediction. arXiv preprint arXiv:1904.08375, 2019.","title":"Document expansion by query prediction","authors":"Nogueira","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Robertson and Walker, 1994] Stephen E Robertson and Steve Walker. Some simple effective approximations to the 2-poisson model for probabilistic weighted retrieval. In SIGIR'94, pages 232-241. Springer, 1994.","title":"Some simple effective approximations to the 2-poisson model for probabilistic weighted retrieval","authors":"E Stephen, Steve Robertson, Walker","year":"1994","venue":"SIGIR'94","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Xiong et al., 2021] Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. Approximate nearest neighbor negative contrastive learning for dense text re- trieval. In International Conference on Learning Repre- sentations, 2021.","title":"Approximate nearest neighbor negative contrastive learning for dense text retrieval","authors":"Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N Bennett, Junaid Ahmed, Arnold Overwijk","year":"2021","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Zhan et al., 2020] Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Min Zhang, and Shaoping Ma. Repbert: Contextualized text embeddings for first-stage retrieval. arXiv preprint arXiv:2006.15498, 2020.","title":"Repbert: Contextualized text embeddings for first-stage retrieval","authors":"Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Min Zhang, Shaoping Ma","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Zhan et al., 2021a] Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Ji- afeng Guo, Min Zhang, and Shaoping Ma. Jointly opti- mizing query encoder and product quantization to improve retrieval performance. In Proceedings of the 30th ACM International Conference on Information and Knowledge Management, 2021.","title":"Jointly optimizing query encoder and product quantization to improve retrieval performance","authors":"Zhan","year":"2021","venue":"Proceedings of the 30th ACM International Conference on Information and Knowledge Management","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zhan et al., 2021b] Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Jiafeng Guo, Min Zhang, and Shaoping Ma. Optimizing dense retrieval model training with hard negatives. In Pro- ceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '21, page 1503-1512, 2021.","title":"Optimizing dense retrieval model training with hard negatives","authors":"Zhan","year":"2021","venue":"Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '21","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhan et al., 2022] Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Ji- afeng Guo, Min Zhang, and Shaoping Ma. Learning dis- crete representations via constrained clustering for effec- tive and efficient dense retrieval. In Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining, WSDM '22, page 1328-1336. Associa- tion for Computing Machinery, 2022.","title":"Learning discrete representations via constrained clustering for effective and efficient dense retrieval","authors":"Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Jiafeng Guo, Min Zhang, Shaoping Ma","year":"2022","venue":"Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining, WSDM '22","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}