{"type":"ScholarlyArticle","title":"Approximating Word Ranking and Negative Sampling for Word Embedding","authors":[{"name":"Guibing Guo","slug":"guibing-guo","given":"Guibing","family":"Guo","affiliation":"# Northeastern University , China"},{"name":"Shichang Ouyang","slug":"shichang-ouyang","given":"Shichang","family":"Ouyang","affiliation":"# Northeastern University , China"},{"name":"Fajie Yuan","slug":"fajie-yuan","given":"Fajie","family":"Yuan","affiliation":"† University of Glasgow , UK"},{"name":"Xingwei Wang","slug":"xingwei-wang","given":"Xingwei","family":"Wang","affiliation":""}],"abstract":"CBOW (Continuous Bag-Of-Words) is  one of the most commonly used\r\ntechniques to generate word embeddings in various NLP tasks. However, it fails to reach the optimal performance due to uniform involvements of positive words and a simple sampling distribution of negative words. To resolve these issues, we propose OptRank to optimize word ranking and approximate negative sampling for bettering word embedding. Specifically, we first formalize word embedding as a ranking problem. Then, we weigh the positive words by their ranks such that highly ranked words have more importance, and adopt a dynamic sampling strategy to select informative negative words. In addition, an approximation method is designed to efficiently compute word ranks. Empirical experiments show that OptRank consistently outperforms its counterparts on a benchmark dataset with different sampling scales, especially when the sampled subset is small. The code and datasets can be obtained from https://github.com/ouououououou/OptRank.","doi":"10.24963/ijcai.2018/569","url":"https://ijcai.org/proceedings/2018/569","pdf":"https://ijcai.org/proceedings/2018/0569.pdf","markdown":"https://ijcai.org/proceedings/2018/569.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/569","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Natural Language Processing","track":"Main track","number":569,"pages":{"first":4092,"last":4098},"keywords":["Natural Language Processing: Natural Language Semantics","Natural Language Processing: Embeddings"],"videos":[],"references":[{"position":1,"raw":"[Agirre et al., 2009] Eneko Agirre, Enrique Alfonseca, Kei- th Hall, Jana Kravalova, MariusPasca, and Aitor Soroa. A study on similarity and relatedness using distributional and wordnet-based approaches. In Proceedings of Human Lan- guage Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Compu- tational Linguistics, pages 19-27, Stroudsburg, PA, USA, 2009. Association for Computational Linguistics.","title":"A study on similarity and relatedness using distributional and wordnet-based approaches","authors":"Eneko Agirre, Enrique Alfonseca, Keith Hall, Jana Kravalova, Aitor Mariuspasca, Soroa","year":"2009","venue":"Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bruni et al., 2012] Elia Bruni, Gemma Boleda, Marco Ba- roni, and Nam-Khanh Tran. Distributional semantics in technicolor. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Pa- pers -Volume 1, pages 136-145, Stroudsburg, PA, USA, 2012. Association for Computational Linguistics.","title":"Distributional semantics in technicolor","authors":"Elia Bruni, Gemma Boleda, Marco Baroni, Nam-Khanh Tran","year":"2012","venue":"Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2017] Long Chen, Fajie Yuan, Joemon M. Jose, and Weinan Zhang. Improving negative sampling for word representation using self-embedded features. CoRR, ab- s/1710.0980, 2017.","title":"Improving negative sampling for word representation using self-embedded features","authors":"Long Chen, Fajie Yuan, Joemon M Jose, Weinan Zhang","year":"2017","venue":"CoRR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Finkelstein et al., 2001] Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. Placing search in context: The concept revisited. In Proceedings of the 10th International Conference on World Wide Web, pages 406-414, New York, NY, USA, 2001. ACM.","title":"Placing search in context: The concept revisited","authors":"Finkelstein","year":"2001","venue":"Proceedings of the 10th International Conference on World Wide Web","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Hill et al., 2015] Felix Hill, Roi Reichart, and Anna Korho- nen. Simlex-999: Evaluating semantic models with (gen- uine) similarity estimation. Computational Linguistics, 41(4):665-695, 2015.","title":"Simlex-999: Evaluating semantic models with (genuine) similarity estimation","authors":"Felix Hill, Roi Reichart, Anna Korhonen","year":"2015","venue":"Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Ji et al., 2015] Shihao Ji, Hyokun Yun, Pinar Yanardag, Shin Matsushima, and S. V. N. Vishwanathan. Wordrank: Learning word embeddings via robust ranking. CoRR, ab- s/1506.02761, 2015.","title":"Wordrank: Learning word embeddings via robust ranking","authors":"Shihao Ji, Hyokun Yun, Pinar Yanardag, Shin Matsushima, S V N Vishwanathan","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Le and Mikolov, 2014] Quoc Le and Tomas Mikolov. Dis- tributed representations of sentences and documents. In Proceedings of the 31st International Conference on Ma- chine Learning, volume 32, pages 1188-1196, Bejing, China, 22-24 Jun 2014. PMLR.","title":"Quoc Le and Tomas Mikolov. Distributed representations of sentences and documents","authors":"Mikolov Le","year":"2014","venue":"Proceedings of the 31st International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Mikolov et al., 2013] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed repre- sentations of words and phrases and their compositionali- ty. In Advances in Neural Information Processing Systems 26, pages 3111-3119.","title":"Distributed representations of words and phrases and their compositionality","authors":"Mikolov","year":"2013","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"Curran Associates, Inc., 2013.","title":null,"authors":"Curran Associates, Inc","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Mnih and Hinton, 2009] Andriy Mnih and Geoffrey E Hin- ton. A scalable hierarchical distributed language model. In Advances in Neural Information Processing Systems 21, pages 1081-1088.","title":"A scalable hierarchical distributed language model","authors":"Andriy Mnih, Geoffrey E Hinton","year":null,"venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"Curran Associates, Inc., 2009.","title":null,"authors":"Curran Associates, Inc","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Pennington et al., 2014] Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), volume 14, pages 1532 - 1543, 2014.","title":"Glove: Global vectors for word representation","authors":"Jeffrey Pennington, Richard Socher, Christopher D Manning","year":"2014","venue":"Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Radinsky et al., 2011] Kira Radinsky, Eugene Agichtein, Evgeniy Gabrilovich, and Shaul Markovitch. A word at a time: Computing word relatedness using temporal se- mantic analysis. In Proceedings of the 20th International Conference on World Wide Web, pages 337-346, New Y- ork, NY, USA, 2011. ACM.","title":"A word at a time: Computing word relatedness using temporal semantic analysis","authors":"Kira Radinsky, Eugene Agichtein, Evgeniy Gabrilovich, Shaul Markovitch","year":"2011","venue":"Proceedings of the 20th International Conference on World Wide Web","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Weston et al., 2010] Jason Weston, Samy Bengio, and Nicolas Usunier. Large scale image annotation: learning to rank with joint word-image embeddings. Machine learn- ing, 81(1):21-35, 2010.","title":"Large scale image annotation: learning to rank with joint word-image embeddings","authors":"Jason Weston, Samy Bengio, Nicolas Usunier","year":"2010","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Xin et al., 2018] xin Xin, Fajie Yuan, Xiangnan He, and Joemon Jose. Batch is not heavy: Learning word embed- dings from all samples. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, 2018.","title":"Batch is not heavy: Learning word embeddings from all samples","authors":"Xin","year":"2018","venue":"Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Yuan et al., 2016] Fajie Yuan, Guibing Guo, Joemon M. Jose, Long Chen, Haitao Yu, and Weinan Zhang. Lambdafm: Learning optimal ranking with factorization machines using lambda surrogates. In Proceedings of the 25th ACM International on Conference on Information and Knowledge Management, pages 227-236, New York, NY, USA, 2016. ACM.","title":"Lambdafm: Learning optimal ranking with factorization machines using lambda surrogates","authors":"Yuan","year":"2016","venue":"Proceedings of the 25th ACM International on Conference on Information and Knowledge Management","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}