{"type":"ScholarlyArticle","title":"Complementary Learning of Word Embeddings","authors":[{"name":"Yan Song","slug":"yan-song","given":"Yan","family":"Song","affiliation":"Tencent AI Lab"},{"name":"Shuming Shi","slug":"shuming-shi","given":"Shuming","family":"Shi","affiliation":"Tencent AI Lab"}],"abstract":"Continuous bag-of-words (CB) and skip-gram (SG) models are popular approaches to training word embeddings. Conventionally they are two standing-alone techniques used individually. However, with the same goal of building embeddings by leveraging surrounding words, they are in fact a pair of complementary tasks where the output of one model can be used as input of the other, and vice versa. In this paper, we propose complementary learning of word embeddings based on the CB and SG model. Specifically, one round of learning first integrates the predicted output of a SG model with existing context, then forms an enlarged context as input to the CB model. Final models are obtained through several rounds of parameter updating. Experimental results indicate that our approach can effectively improve the quality of initial embeddings, in terms of intrinsic and extrinsic evaluations.","doi":"10.24963/ijcai.2018/607","url":"https://ijcai.org/proceedings/2018/607","pdf":"https://ijcai.org/proceedings/2018/0607.pdf","markdown":"https://ijcai.org/proceedings/2018/607.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/607","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Natural Language Processing","track":"Main track","number":607,"pages":{"first":4368,"last":4374},"keywords":["Natural Language Processing: Natural Language Processing","Natural Language Processing: Embeddings"],"videos":[],"references":[{"position":1,"raw":"[Bruni et al., 2012] Elia Bruni, Gemma Boleda, Marco Ba- roni, and Nam Khanh Tran. Distributional Semantics in Technicolor. In Proceedings of ACL, pages 136-145, Jeju Island, Korea, July 2012.","title":"Distributional Semantics in Technicolor","authors":"Elia Bruni, Gemma Boleda, Marco Baroni, Nam Khanh, Tran","year":"2012","venue":"Proceedings of ACL","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Collobert and Weston, 2008] Ronan Collobert and Jason Weston. A Unified Architecture for Natural Language Pro- cessing: Deep Neural Networks with Multitask Learning. In Proceedings of ICML, pages 160-167, New York, NY, USA, 2008. ACM.","title":"A Unified Architecture for Natural Language Processing: Deep Neural Networks with Multitask Learning","authors":"Weston ; Collobert, Ronan Collobert, Jason Weston","year":"2008","venue":"Proceedings of ICML","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Collobert et al., 2011] Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. Natural Language Processing (Almost) from Scratch. Journal of Machine Learning Research, 12:2493- 2537, Nov 2011.","title":"Natural Language Processing (Almost) from Scratch","authors":"Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, Pavel Kuksa","year":"2011","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dai and Le, 2015] Andrew M. Dai and Quoc V. Le. Semi- supervised Sequence Learning. In NIPS, pages 3079- 3087, December 2015.","title":"Semisupervised Sequence Learning","authors":"Andrew M Dai, V Quoc, Le","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Faruqui et al., 2015] Manaal Faruqui, Jesse Dodge, Su- jay Kumar Jauhar, Chris Dyer, Eduard Hovy, and Noah A. Smith. Retrofitting Word Vectors to Semantic Lexicons. In Proceedings of NAACL-HLT, pages 1606-1615, Den- ver, Colorado, May-June 2015.","title":"Retrofitting Word Vectors to Semantic Lexicons","authors":"Faruqui","year":"2015","venue":"Proceedings of NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Finkelstein et al., 2002] Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. Placing Search in Context: the Concept Revisited. ACM Transaction on Information Systems, 20(1):116-131, 2002.","title":"Placing Search in Context: the Concept Revisited","authors":"Finkelstein","year":"2002","venue":"ACM Transaction on Information Systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Ganitkevitch et al., 2013] Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. PPDB: The Paraphrase Database. In Proceedings of NAACL-HLT, pages 758-764, Atlanta, Georgia, June 2013.","title":"PPDB: The Paraphrase Database","authors":"Juri Ganitkevitch, Benjamin Van Durme, Chris Callison-Burch","year":"2013","venue":"Proceedings of NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hemphill et al., 1990] Charles T. Hemphill, John J. God- frey, and George R. Doddington. The ATIS Spoken Lan- guage Systems Pilot Corpus. In Proceedings of the Work- shop on Speech and Natural Language, HLT '90, pages 96-101, 1990.","title":"The ATIS Spoken Language Systems Pilot Corpus","authors":"Charles T Hemphill, John J Godfrey, George R Doddington","year":"1990","venue":"Proceedings of the Workshop on Speech and Natural Language, HLT '90","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hill et al., 2015] Felix Hill, Roi Reichart, and Anna Korho- nen. Simlex-999: Evaluating Semantic Models with Gen- uine Similarity Estimation. Computational Linguistics, 41(4):665-695, December 2015.","title":"Felix Hill, Roi Reichart, and Anna Korhonen. Simlex-999: Evaluating Semantic Models with Genuine Similarity Estimation","authors":"Hill","year":"2015","venue":"Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Iacobacci et al., 2015] Ignacio Iacobacci, Moham- mad Taher Pilehvar, and Roberto Navigli. SensEmbed: Learning Sense Embeddings for Word and Relational Similarity. In Proceedings of ACL, pages 95-105, Beijing, China, July 2015.","title":"SensEmbed: Learning Sense Embeddings for Word and Relational Similarity","authors":"Ignacio Iacobacci, Mohammad Taher Pilehvar, Roberto Navigli","year":"2015","venue":"Proceedings of ACL","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Joulin et al., 2016] Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. Bag of Tricks for Effi- cient Text Classification. arXiv preprint, abs/1607.01759, 2016.","title":"Bag of Tricks for Efficient Text Classification","authors":"Armand Joulin, Edouard Grave, Piotr Bojanowski, Tomas Mikolov","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kiela et al., 2015] Douwe Kiela, Felix Hill, and Stephen Clark. Specializing Word Embeddings for Similarity or Relatedness. In Proceedings of EMNLP, pages 2044- 2048, Lisbon, Portugal, September 2015.","title":"Specializing Word Embeddings for Similarity or Relatedness","authors":"Kiela","year":"2015","venue":"Proceedings of EMNLP","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li and Roth, 2002] Xin Li and Dan Roth. Learning Ques- tion Classifiers. In COLING, pages 1-7, 2002.","title":"Learning Question Classifiers","authors":"Roth ; Li, Xin Li, Dan Roth","year":"2002","venue":"COLING","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Maas et al., 2011] Andrew L. Maas, Raymond E. Daly, Pe- ter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning Word Vectors for Sentiment Analysis. In Proceedings of ACL, pages 142-150, Portland, Oregon, USA, June 2011.","title":"Learning Word Vectors for Sentiment Analysis","authors":"Andrew L Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, Christopher Potts","year":"2011","venue":"Proceedings of ACL","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mikolov et al., 2010] Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernocký, and Sanjeev Khudanpur. Re- current Neural Network based Language Model. In IN- TERSPEECH, pages 1045-1048, 2010.","title":"Recurrent Neural Network based Language Model","authors":"Mikolov","year":"2010","venue":"IN-TERSPEECH","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mikolov et al., 2013a] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient Estimation of Word Representations in Vector Space. arXiv preprint, abs/1301.3781, 2013.","title":"Efficient Estimation of Word Representations in Vector Space","authors":"Mikolov","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Mikolov et al., 2013b] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed Repre- sentations of Words and Phrases and their Compositional- ity. In NIPS, pages 3111-3119. 2013.","title":"Distributed Representations of Words and Phrases and their Compositionality","authors":"Mikolov","year":"2013","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Pennington et al., 2014] Jeffrey Pennington, Richard Socher, and Christopher Manning. GloVe: Global Vectors for Word Representation. In Proceedings of EMNLP, pages 1532-1543, Doha, Qatar, October 2014.","title":"GloVe: Global Vectors for Word Representation","authors":"Jeffrey Pennington, Richard Socher, Christopher Manning","year":"2014","venue":"Proceedings of EMNLP","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Song et al., 2017] Yan Song, Chia-Jung Lee, and Fei Xia. Learning Word Representations with Regularization from Prior Knowledge. In Proceedings of CoNLL, pages 143- 152, Vancouver, Canada, August 2017.","title":"Learning Word Representations with Regularization from Prior Knowledge","authors":"Yan Song, Chia-Jung Lee, Fei Xia","year":"2017","venue":"Proceedings of CoNLL","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Sutton and Barto, 1998] Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. MIT Press, 1998.","title":"Reinforcement Learning: An Introduction","authors":"Richard S Sutton, Andrew G Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sutton et al., 2000] Richard S. Sutton, David Mcallester, Satinder Singh, and Yishay Mansour. Policy Gradient Methods for Reinforcement Learning with Function Ap- proximation. In NIPS, pages 1057-1063. MIT Press, 2000.","title":"Policy Gradient Methods for Reinforcement Learning with Function Approximation","authors":"Richard S Sutton, David Mcallester, Satinder Singh, Yishay Mansour","year":"2000","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Yu and Dredze, 2014] Mo Yu and Mark Dredze. Improving Lexical Embeddings with Semantic Knowledge. In ACL, pages 545-550, Baltimore, Maryland, June 2014.","title":"Improving Lexical Embeddings with Semantic Knowledge","authors":"Mo Yu, Mark Dredze","year":"2014","venue":"ACL","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}