{"type":"ScholarlyArticle","title":"Mutual Information Estimation using LSH Sampling","authors":[{"name":"Ryan Spring","slug":"ryan-spring","given":"Ryan","family":"Spring","affiliation":"Rice University , Houston , Texas , USA"},{"name":"Anshumali Shrivastava","slug":"anshumali-shrivastava","given":"Anshumali","family":"Shrivastava","affiliation":"Rice University , Houston , Texas , USA"}],"abstract":"Learning representations in an unsupervised or self-supervised manner is a growing area of research. Current approaches in representation learning seek to maximize the mutual information between the learned representation and original data.  One of the most popular ways to estimate mutual information (MI) is based on Noise Contrastive Estimation (NCE). This MI estimate exhibits low variance, but it is upper-bounded by log(N), where N is the number of samples. \r\n\r\nIn an ideal scenario, we would use the entire dataset to get the most accurate estimate. However, using such a large number of samples is computationally prohibitive. Our proposed solution is to decouple the upper-bound for the MI estimate from the sample size. Instead, we estimate the partition function of the NCE loss function for the entire dataset using importance sampling (IS). In this paper, we use locality-sensitive hashing (LSH) as an adaptive sampler and propose an unbiased estimator that accurately approximates the partition function in sub-linear (near-constant) time. The samples are correlated and non-normalized, but the derived estimator is unbiased without any assumptions. We show that our LSH sampling estimate provides a superior bias-variance trade-off when compared to other state-of-the-art approaches.","doi":"10.24963/ijcai.2020/389","url":"https://ijcai.org/proceedings/2020/389","pdf":"https://ijcai.org/proceedings/2020/0389.pdf","markdown":"https://ijcai.org/proceedings/2020/389.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/389","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":389,"pages":{"first":2807,"last":2815},"keywords":["Machine Learning: Bayesian Optimization","Data Mining: Big Data, Large-Scale Systems","Machine Learning: Big data; Scalability","Data Mining: Clustering, Unsupervised Learning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24984/file_24984.m3u8","length_sec":282,"provider":"hls"}],"references":[{"position":1,"raw":"[Andoni and Indyk, 2004] Alexandr Andoni and Piotr Indyk. E2lsh: Exact euclidean locality sensitive hashing. Techni- cal report, MIT, 2004.","title":"Alexandr Andoni and Piotr Indyk. E2lsh: Exact euclidean locality sensitive hashing","authors":"Indyk Andoni","year":"2004","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bachman et al., 2019] Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. In Advances in Neural Information Processing Systems, pages 15509- 15519, 2019.","title":"Learning representations by maximizing mutual information across views","authors":"Philip Bachman, Devon Hjelm, William Buchwalter","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bell and Sejnowski, 1995] Anthony J Bell and Terrence J Sejnowski. An information-maximization approach to blind separation and blind deconvolution. Neural compu- tation, 7(6):1129-1159, 1995.","title":"An information-maximization approach to blind separation and blind deconvolution","authors":"J Anthony, Terrence J Bell, Sejnowski","year":"1995","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Charikar and Siminelakis, 2017] Moses Charikar and Paris Siminelakis. Hashing-based-estimators for kernel density in high dimensions. In 2017 IEEE 58th Annual Sympo- sium on Foundations of Computer Science (FOCS), pages 1032-1043. IEEE, 2017.","title":"Moses Charikar and Paris Siminelakis. Hashing-based-estimators for kernel density in high dimensions","authors":"Siminelakis Charikar","year":"2017","venue":"IEEE 58th Annual Symposium on Foundations of Computer Science (FOCS)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Charikar, 2002] Moses S Charikar. Similarity estimation techniques from rounding algorithms. In Proceedings of the thiry-fourth annual ACM symposium on Theory of computing, pages 380-388. ACM, 2002.","title":"Similarity estimation techniques from rounding algorithms","authors":"Moses S Charikar, Charikar","year":"2002","venue":"Proceedings of the thiry-fourth annual ACM symposium on Theory of computing","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen et al., 2019] Beidi Chen, Yingchen Xu, and Anshu- mali Shrivastava. Fast and accurate stochastic gradient es- timation. In Advances in Neural Information Processing Systems, pages 12339-12349, 2019.","title":"Fast and accurate stochastic gradient estimation","authors":"Chen","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Coleman and Shrivastava, 2019] Benjamin Coleman and Anshumali Shrivastava. Sub-linear race sketches for ap- proximate kernel density estimation on streaming data. arXiv preprint arXiv:1912.02283, 2019.","title":"Sub-linear race sketches for approximate kernel density estimation on streaming data","authors":"Coleman, Shrivastava, Benjamin Coleman, Anshumali Shrivastava","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Devlin et al., 2018] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint arXiv:1810.04805, 2018.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gionis et al., 1999] Aristides Gionis, Piotr Indyk, and Ra- jeev Motwani. Similarity search in high dimensions via hashing. VLDB, 99(6):518-529, 1999.","title":"Similarity search in high dimensions via hashing","authors":"Gionis","year":"1999","venue":"VLDB","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gutmann and Hyvärinen, 2010] Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pages 297-304, 2010.","title":"Noise-contrastive estimation: A new estimation principle for unnormalized statistical models","authors":"Hyvärinen Gutmann, Michael Gutmann, Aapo Hyvärinen","year":"2010","venue":"Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[He et al., 2012] Junfeng He, Sanjiv Kumar, and Shih-fu Chang. On the difficulty of nearest neighbor search. In Proceedings of the 29th International Conference on Ma- chine Learning (ICML-12), pages 1127-1134, 2012.","title":"On the difficulty of nearest neighbor search","authors":"Junfeng He, Sanjiv Kumar, Shih-Fu Chang","year":"2012","venue":"Proceedings of the 29th International Conference on Machine Learning (ICML-12)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hjelm et al., 2018] R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representa- tions by mutual information estimation and maximization. arXiv preprint arXiv:1808.06670, 2018.","title":"Learning deep representations by mutual information estimation and maximization","authors":"Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, Yoshua Bengio","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Indyk and Motwani, 1998] Piotr Indyk and Rajeev Mot- wani. Approximate nearest neighbors: Towards remov- ing the curse of dimensionality. In STOC, pages 604-613, Dallas, TX, 1998.","title":"Approximate nearest neighbors: Towards remov-ing the curse of dimensionality","authors":"Piotr Indyk, Rajeev Motwani","year":"1998","venue":"STOC","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Leskovec et al., 2014] Jure Leskovec, Anand Rajaraman, and Jeffrey David Ullman. Mining of massive datasets. Cambridge University Press, 2014.","title":"Mining of massive datasets","authors":"Jure Leskovec, Anand Rajaraman, Jeffrey David Ullman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Linsker, 1988] Ralph Linsker. Self-organization in a per- ceptual network. Computer, 21(3):105-117, 1988.","title":"Ralph Linsker. Self-organization in a perceptual network","authors":"Linsker","year":"1988","venue":"Computer","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liu et al., 2015] Qiang Liu, Jian Peng, Alexander Ihler, and John Fisher III. Estimating the partition function by dis- criminance sampling. In Proceedings of the Thirty-First Conference on Uncertainty in Artificial Intelligence, pages 514-522. AUAI Press, 2015.","title":"Estimating the partition function by discriminance sampling","authors":"Qiang Liu, Jian Peng, Alexander Ihler, John Fisher, Iii","year":"2015","venue":"Proceedings of the Thirty-First Conference on Uncertainty in Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Luo and Shrivastava, 2018] Chen Luo and Anshumali Shri- vastava. Arrays of (locality-sensitive) count estimators (ace) anomaly detection on the edge. In Proceedings of the 2018 World Wide Web Conference, pages 1439-1448, 2018.","title":"Arrays of (locality-sensitive) count estimators (ace) anomaly detection on the edge","authors":"Chen Luo, Anshumali Shrivastava","year":"2018","venue":"Proceedings of the 2018 World Wide Web Conference","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Luo and Shrivastava, 2019] Chen Luo and Anshumali Shri- vastava. Scaling-up split-merge mcmc with locality sensi- tive sampling (lss). In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 4464-4471, 2019.","title":"Scaling-up split-merge mcmc with locality sensitive sampling (lss)","authors":"Chen Luo, Anshumali Shrivastava","year":"2019","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Mussmann et al., 2017] Stephen Mussmann, Daniel Levy, and Stefano Ermon. Fast amortized inference and learn- ing in log-linear models with randomly perturbed near- est neighbor search. In Proceedings of the Thirty-Third Conference on Uncertainty in Artificial Intelligence. AUAI Press, 2017.","title":"Fast amortized inference and learning in log-linear models with randomly perturbed nearest neighbor search","authors":"Stephen Mussmann, Daniel Levy, Stefano Ermon","year":"2017","venue":"Proceedings of the Thirty-Third Conference on Uncertainty in Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Nguyen et al., 2010] XuanLong Nguyen, Martin J Wain- wright, and Michael I Jordan. Estimating divergence functionals and the likelihood ratio by convex risk min- imization. IEEE Transactions on Information Theory, 56(11):5847-5861, 2010.","title":"Estimating divergence functionals and the likelihood ratio by convex risk minimization","authors":"Nguyen","year":"2010","venue":"IEEE Transactions on Information Theory","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Oord et al., 2018a] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018.","title":"Representation learning with contrastive predictive coding","authors":"Oord","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Oord et al., 2018b] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018.","title":"Representation learning with contrastive predictive coding","authors":"Oord","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Poole et al., 2019] Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On varia- tional bounds of mutual information. In Proceedings of the 36th International Conference on Machine Learning, pages 5171-5180, 2019.","title":"On variational bounds of mutual information","authors":"Ben Poole, Sherjil Ozair, Aaron Van Den, Alex Oord, George Alemi, Tucker","year":"2019","venue":"Proceedings of the 36th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Spring and Shrivastava, 2017a] Ryan Spring and Anshu- mali Shrivastava. A new unbiased and efficient class of lsh-based samplers and estimators for partition func- tion computation in log-linear models. arXiv preprint arXiv:1703.05160, 2017.","title":"Ryan Spring and Anshumali Shrivastava. A new unbiased and efficient class of lsh-based samplers and estimators for partition function computation in log-linear models","authors":"Shrivastava Spring","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Spring and Shrivastava, 2017b] Ryan Spring and Anshu- mali Shrivastava. Scalable and sustainable deep learning via randomized hashing. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discov- ery and Data Mining, pages 445-454, 2017.","title":"Ryan Spring and Anshumali Shrivastava. Scalable and sustainable deep learning via randomized hashing","authors":"Shrivastava Spring","year":"2017","venue":"Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}