{"type":"ScholarlyArticle","title":"Rethinking InfoNCE: How Many Negative Samples Do You Need?","authors":[{"name":"Chuhan Wu","slug":"chuhan-wu","given":"Chuhan","family":"Wu","affiliation":"Department of Electronic Engineering , Tsinghua University , Beijing 100084 , China"},{"name":"Fangzhao Wu","slug":"fangzhao-wu","given":"Fangzhao","family":"Wu","affiliation":"Microsoft Research Asia , Beijing 100080 , China"},{"name":"Yongfeng Huang","slug":"yongfeng-huang","given":"Yongfeng","family":"Huang","affiliation":"Department of Electronic Engineering , Tsinghua University , Beijing 100084 , China"}],"abstract":"InfoNCE is a widely used contrastive training loss. It aims to estimate the mutual information between a pair of variables by discriminating between each positive pair and its associated K negative pairs. It is proved that when the sample labels are clean, the lower bound of mutual information estimation is tighter when more negative samples are incorporated, which usually yields better model performance. However, in practice the labels often contain noise, and incorporating too many noisy negative samples into model training may be suboptimal. In this paper, we study how many negative samples are optimal for InfoNCE in different scenarios via a semi-quantitative theoretical framework. More specifically, we first propose a probabilistic model to analyze the influence of the negative sampling ratio K on training sample informativeness. Then, we design a training effectiveness function to measure the overall influence of training samples based on their informativeness. We estimate the optimal negative sampling ratio using the K value that maximizes the training effectiveness function. Based on our framework, we further propose an adaptive negative sampling method that can dynamically adjust the negative sampling ratio to improve InfoNCE-based model training. Extensive experiments in three different tasks show our framework can accurately predict the optimal negative sampling ratio, and various models can benefit from our adaptive negative sampling method.","doi":"10.24963/ijcai.2022/348","url":"https://ijcai.org/proceedings/2022/348","pdf":"https://ijcai.org/proceedings/2022/0348.pdf","markdown":"https://ijcai.org/proceedings/2022/348.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/348","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Humans and AI","track":"Main Track","number":348,"pages":{"first":2509,"last":2515},"keywords":["Humans and AI: Personalization and User Modeling","Data Mining: Information Retrieval"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986952","length_sec":11,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985095","length_sec":341,"provider":"slideslive"}],"references":[{"position":1,"raw":"[An et al., 2019] Mingxiao An, Fangzhao Wu, Chuhan Wu, Kun Zhang, Zheng Liu, and Xing Xie. Neural news recom- mendation with long-and short-term user representations. In ACL, pages 336-345, 2019.","title":"Neural news recommendation with long-and short-term user representations","authors":"An","year":"2019","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Arora et al., 2019] Sanjeev Arora, Hrishikesh Khande- parkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saunshi. A theoretical analysis of contrastive unsupervised representation learning. In ICML, pages 9904-9923, 2019.","title":"A theoretical analysis of contrastive unsupervised representation learning","authors":"Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, Nikunj Saunshi","year":"2019","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Behrmann et al., 2020] Nadine Behrmann, Jurgen Gall, and Mehdi Noroozi. Unsupervised video representation learn- ing by bidirectional feature prediction. In CVPR, pages 1670-1679, 2020.","title":"Unsupervised video representation learning by bidirectional feature prediction","authors":"Nadine Behrmann, Jurgen Gall, Mehdi Noroozi","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chang et al., 2020] Wei-Cheng Chang, Felix X Yu, Yin- Wen Chang, Yiming Yang, and Sanjiv Kumar. Pre-training tasks for embedding-based large-scale retrieval. arXiv preprint arXiv:2002.03932, 2020.","title":"Pre-training tasks for embedding-based large-scale retrieval","authors":"Wei-Cheng Chang, Felix X Yu, Yin-Wen Chang, Yiming Yang, Sanjiv Kumar","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chi et al., 2021] Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian- Ling Mao, Heyan Huang, and Ming Zhou. Infoxlm: An information-theoretic framework for cross-lingual lan- guage model pre-training. 2021.","title":"Infoxlm: An information-theoretic framework for cross-lingual language model pre-training","authors":"Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, Ming Zhou","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Goyal et al., 2017] Priya Goyal, Piotr Dollár, Ross Gir- shick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Ky- rola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Ac- curate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677, 2017.","title":"Accurate, large minibatch sgd: Training imagenet in 1 hour","authors":"Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, Kaiming He","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Harper and Konstan, 2015] F Maxwell Harper and Joseph A Konstan. The movielens datasets: History and context. TIIS, 5(4):1-19, 2015.","title":"The movielens datasets: History and context","authors":"Maxwell Harper, Joseph A Konstan","year":"2015","venue":"TIIS","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[He et al., 2020] Kaiming He, Haoqi Fan, Yuxin Wu, Sain- ing Xie, and Ross Girshick. Momentum contrast for un- supervised visual representation learning. In CVPR, pages 9729-9738, 2020.","title":"Momentum contrast for unsupervised visual representation learning","authors":"Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, Ross Girshick","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hjelm et al., 2019] R. Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Philip Bachman, Adam Trischler, and Yoshua Bengio. Learning deep rep- resentations by mutual information estimation and maxi- mization. In ICLR, 2019.","title":"Learning deep representations by mutual information estimation and maximization","authors":"R, Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Philip Bachman, Adam Trischler, Yoshua Bengio","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Huang et al., 2013] Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. Learning deep structured semantic models for web search using click- through data. In CIKM, pages 2333-2338, 2013.","title":"Learning deep structured semantic models for web search using clickthrough data","authors":"Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, Larry Heck","year":"2013","venue":"CIKM","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kalantidis et al., 2020] Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. Hard negative mixing for contrastive learning. arXiv preprint arXiv:2010.01028, 2020.","title":"Hard negative mixing for contrastive learning","authors":"Yannis Kalantidis, Bulent Mert, Noe Sariyildiz, Philippe Pion, Diane Weinzaepfel, Larlus","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kingma and Ba, 2015] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Menon et al., 2019] Aditya Krishna Menon, Ankit Singh Rawat, Sashank J Reddi, and Sanjiv Kumar. Can gradi- ent clipping mitigate label noise? In ICLR, 2019.","title":"Can gradient clipping mitigate label noise?","authors":"Aditya Krishna Menon, Ankit Singh Rawat, Sashank J Reddi, Sanjiv Kumar","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Natarajan et al., 2013] Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learn- ing with noisy labels. NIPS, 26:1196-1204, 2013.","title":"Learning with noisy labels","authors":"Nagarajan Natarajan, Pradeep K Inderjit S Dhillon, Ambuj Ravikumar, Tewari","year":"2013","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Oord et al., 2018] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv:1807.03748, 2018.","title":"Representation learning with contrastive predictive coding","authors":"Aaron Van Den Oord, Yazhe Li, Oriol Vinyals","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Reimers and Gurevych, 2019] Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In EMNLP-IJCNLP, pages 3973-3983, 2019.","title":"Sentence-bert: Sentence embeddings using siamese bert-networks","authors":"Gurevych Reimers, Nils Reimers, Iryna Gurevych","year":"2019","venue":"EMNLP-IJCNLP","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Rendle et al., 2009] Steffen Rendle, Christoph Freuden- thaler, Zeno Gantner, and Lars Schmidt-Thieme. Bpr: Bayesian personalized ranking from implicit feedback. In UAI, pages 452-461, 2009.","title":"Bpr: Bayesian personalized ranking from implicit feedback","authors":"Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, Lars Schmidt-Thieme","year":"2009","venue":"UAI","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Robinson et al., 2020] Joshua Robinson, Ching-Yao Chuang, Suvrit Sra, and Stefanie Jegelka. Contrastive learning with hard negative samples. arXiv preprint arXiv:2010.04592, 2020.","title":"Contrastive learning with hard negative samples","authors":"Joshua Robinson, Ching-Yao Chuang, Suvrit Sra, Stefanie Jegelka","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Shao et al., 2020] Jie Shao, Xin Wen, Bingchen Zhao, Changhu Wang, and Xiangyang Xue. Context encoding for video retrieval with contrastive learning. arXiv preprint arXiv:2008.01334, 2020.","title":"Context encoding for video retrieval with contrastive learning","authors":"Jie Shao, Xin Wen, Bingchen Zhao, Changhu Wang, Xiangyang Xue","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Sun et al., 2019] Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. Bert4rec: Sequen- tial recommendation with bidirectional encoder represen- tations from transformer. In CIKM, pages 1441-1450, 2019.","title":"Bert4rec: Sequential recommendation with bidirectional encoder representations from transformer","authors":"Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, Peng Jiang","year":"2019","venue":"CIKM","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sun et al., 2020] Siqi Sun, Zhe Gan, Yuwei Fang, Yu Cheng, Shuohang Wang, and Jingjing Liu. Con- trastive distillation on intermediate representations for language model compression. In EMNLP, pages 498-508, 2020.","title":"Contrastive distillation on intermediate representations for language model compression","authors":"Siqi Sun, Zhe Gan, Yuwei Fang, Yu Cheng, Shuohang Wang, Jingjing Liu","year":"2020","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wu et al., 2019a] Chuhan Wu, Fangzhao Wu, Mingxiao An, Jianqiang Huang, Yongfeng Huang, and Xing Xie. Neural news recommendation with attentive multi-view learning. In IJCAI, pages 3863-3869, 2019.","title":"Neural news recommendation with attentive multi-view learning","authors":"Wu","year":"2019","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2019/536"},{"position":23,"raw":"[Wu et al., 2019b] Chuhan Wu, Fangzhao Wu, Mingxiao An, Jianqiang Huang, Yongfeng Huang, and Xing Xie. Npa: Neural news recommendation with personalized at- tention. In KDD, pages 2576-2584, 2019.","title":"Npa: Neural news recommendation with personalized attention","authors":"Wu","year":"2019","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wu et al., 2019c] Chuhan Wu, Fangzhao Wu, Suyu Ge, Tao Qi, Yongfeng Huang, and Xing Xie. Neural news rec- ommendation with multi-head self-attention. In EMNLP- IJCNLP, pages 6390-6395, 2019.","title":"Neural news recommendation with multi-head self-attention","authors":"Wu","year":"2019","venue":"EMNLP-IJCNLP","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wu et al., 2020] Fangzhao Wu, Ying Qiao, Jiun-Hung Chen, Chuhan Wu, Tao Qi, Jianxun Lian, Danyang Liu, Xing Xie, Jianfeng Gao, Winnie Wu, et al. Mind: A large- scale dataset for news recommendation. In ACL, pages 3597-3606, 2020.","title":"Mind: A largescale dataset for news recommendation","authors":"Fangzhao Wu, Ying Qiao, Jiun-Hung Chen, Chuhan Wu, Tao Qi, Jianxun Lian, Danyang Liu, Xing Xie, Jianfeng Gao, Winnie Wu","year":"2020","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wu et al., 2021] Jiancan Wu, Xiang Wang, Fuli Feng, Xi- angnan He, Liang Chen, Jianxun Lian, and Xing Xie. Self- supervised graph learning for recommendation. In SIGIR, pages 726-735, 2021.","title":"Selfsupervised graph learning for recommendation","authors":"Jiancan Wu, Xiang Wang, Fuli Feng, Xiangnan He, Liang Chen, Jianxun Lian, Xing Xie","year":"2021","venue":"SIGIR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}