{"type":"ScholarlyArticle","title":"Lifelong Domain Word Embedding via Meta-Learning","authors":[{"name":"Hu Xu","slug":"hu-xu","given":"Hu","family":"Xu","affiliation":"Department of Computer Science , University of Illinois at Chicago , Chicago , IL , USA"},{"name":"Bing Liu","slug":"bing-liu","given":"Bing","family":"Liu","affiliation":"Department of Computer Science , University of Illinois at Chicago , Chicago , IL , USA"},{"name":"Lei Shu","slug":"lei-shu","given":"Lei","family":"Shu","affiliation":"Department of Computer Science , University of Illinois at Chicago , Chicago , IL , USA"},{"name":"Philip S. Yu","slug":"philip-s-yu","given":"Philip S.","family":"Yu","affiliation":"Department of Computer Science , University of Illinois at Chicago , Chicago , IL , USA; Institute for Data Science , Tsinghua University , Beijing , China"}],"abstract":"Learning high-quality domain word embeddings is important for achieving good performance in many NLP tasks. General-purpose embeddings trained on large-scale corpora are often sub-optimal for domain-specific applications. However, domain-specific tasks often do not have large in-domain corpora for training high-quality domain embeddings. In this paper, we propose a novel lifelong learning setting for domain embedding. That is, when performing the new domain embedding, the system has seen many past domains, and it tries to expand the new in-domain corpus by exploiting the corpora from the past domains via meta-learning. The proposed meta-learner characterizes the similarities of the contexts of the same word in many domain corpora, which helps retrieve relevant data from the past domains to expand the new domain corpus. Experimental results show that domain embeddings produced from such a process improve the performance of the downstream tasks.","doi":"10.24963/ijcai.2018/627","url":"https://ijcai.org/proceedings/2018/627","pdf":"https://ijcai.org/proceedings/2018/0627.pdf","markdown":"https://ijcai.org/proceedings/2018/627.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/627","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Natural Language Processing","track":"Main track","number":627,"pages":{"first":4510,"last":4516},"keywords":["Natural Language Processing: Embeddings","Natural Language Processing: Text Classification"],"videos":[],"references":[{"position":1,"raw":"[Andrychowicz et al., 2016] Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, and Nando de Freitas. Learning to learn by gradient descent by gradient descent. In NIPS, pages 3981-3989, 2016.","title":"Learning to learn by gradient descent by gradient descent","authors":"Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Nando De Freitas","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bojanowski et al., 2016] Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vec- tors with subword information. arXiv, 2016.","title":"Enriching word vectors with subword information","authors":"Piotr Bojanowski, Edouard Grave, Armand Joulin, Tomas Mikolov","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bollegala et al., 2015] Danushka Bollegala, Takanori Mae- hara, and Ken-ichi Kawarabayashi. Unsupervised cross- domain word representation learning. In ACL, pages 730- 740, 2015.","title":"Unsupervised crossdomain word representation learning","authors":"Danushka Bollegala, Takanori Maehara, Ken-Ichi Kawarabayashi","year":"2015","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bollegala et al., 2017] Danushka Bollegala, Kohei Hayashi, and Ken-ichi Kawarabayashi. Think globally, em- bed locally-locally linear meta-embedding of words. arXiv:1709.06671, 2017.","title":"Think globally, embed locally-locally linear meta-embedding of words","authors":"Danushka Bollegala, Kohei Hayashi, Ken-Ichi Kawarabayashi","year":"2017","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2018/552"},{"position":5,"raw":"[Chen and Liu, 2014] Zhiyuan Chen and Bing Liu. Topic modeling using topics from many domains, lifelong learn- ing and big data. In ICML, pages 703-711, 2014.","title":"Topic modeling using topics from many domains, lifelong learning and big data","authors":"Chen, Liu ; ] Zhiyuan Chen, Bing Liu","year":"2014","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen and Liu, 2016] Zhiyuan Chen and Bing Liu. Lifelong Machine Learning. Morgan & Claypool Publishers, 2016.","title":"Lifelong Machine Learning","authors":"Zhiyuan Chen, Bing Liu","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Durrett and Klein, 2015] Greg Durrett and Dan Klein. Neu- ral crf parsing. arXiv, 2015.","title":"Neural crf parsing","authors":"Greg Durrett, Dan Klein","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Finn et al., 2017] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML, pages 1126-1135, 2017.","title":"Model-agnostic meta-learning for fast adaptation of deep networks","authors":"Finn","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ganin et al., 2016] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, Franc ¸ois Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. JMLR, pages 2096-2030, 2016.","title":"Domain-adversarial training of neural networks","authors":"Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, Mario Franc ¸ois Laviolette, Victor Marchand, Lempitsky","year":"2016","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[He and McAuley, 2016] R. He and J. McAuley. Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In WWW, pages 507-517, 2016.","title":"Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering","authors":"R He, J Mcauley","year":"2016","venue":"WWW","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-1780, 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kingma and Ba, 2014] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"Diederik Kingma, Jimmy Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Maas et al., 2011] Andrew L Maas, Raymond E Daly, Pe- ter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In ACL, pages 142-150, 2011.","title":"Learning word vectors for sentiment analysis","authors":"Raymond E Andrew L Maas, Peter T Daly, Dan Pham, Andrew Y Huang, Christopher Ng, Potts","year":"2011","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mikolov et al., 2013a] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv:1301.3781, 2013.","title":"Efficient estimation of word representations in vector space","authors":"Mikolov","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mikolov et al., 2013b] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed repre- sentations of words and phrases and their compositionality. In NIPS, pages 3111-3119, 2013.","title":"Distributed representations of words and phrases and their compositionality","authors":"Mikolov","year":"2013","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mikolov et al., 2013c] Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. In hlt-Naacl, pages 746-751, 2013.","title":"Linguistic regularities in continuous space word representations","authors":"Mikolov","year":"2013","venue":"hlt-Naacl","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Mitchell et al., 2018] Tom Mitchell, William Cohen, Este- vam Hruschka, Partha Talukdar, B Yang, J Betteridge, A Carlson, B Dalvi, M Gardner, B Kisiel, et al. Never- ending learning. Communications of the ACM, 61(5):103- 115, 2018.","title":"Neverending learning","authors":"Tom Mitchell, William Cohen, Estevam Hruschka, Partha Talukdar, B Yang, J Betteridge, B Carlson, M Dalvi, B Gardner, Kisiel","year":"2018","venue":"Communications of the ACM","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mnih and Hinton, 2007] Andriy Mnih and Geoffrey Hinton. Three new graphical models for statistical language mod- elling. In ICML, pages 641-648, 2007.","title":"Three new graphical models for statistical language modelling","authors":"Hinton Mnih, Andriy Mnih, Geoffrey Hinton","year":"2007","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Nayak et al., 2016] Neha Nayak, Gabor Angeli, and Christopher D Manning. Evaluating word embeddings using a representative suite of practical tasks. ACL 2016, pages 19-23, 2016.","title":"Evaluating word embeddings using a representative suite of practical tasks","authors":"Neha Nayak, Gabor Angeli, Christopher D Manning","year":"2016","venue":"ACL 2016","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Pan and Yang, 2010] Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE TKDE, pages 1345- 1359, 2010.","title":"A survey on transfer learning","authors":"Jialin Sinno, Qiang Pan, Yang","year":"2010","venue":"IEEE TKDE","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Pennington et al., 2014] Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In EMNLP, volume 14, pages 1532-43, 2014.","title":"Glove: Global vectors for word representation","authors":"Jeffrey Pennington, Richard Socher, Christopher D Manning","year":"2014","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Shu et al., 2016] Lei Shu, Bing Liu, Hu Xu, and Annice Kim. Lifelong-rl: Lifelong relaxation labeling for sepa- rating entities and aspects in opinion targets. In EMNLP, pages 225-235, 2016.","title":"Lifelong-rl: Lifelong relaxation labeling for separating entities and aspects in opinion targets","authors":"Lei Shu, Bing Liu, Hu Xu, Annice Kim","year":"2016","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sienčnik, 2015] Scharolta Katharina Sienčnik. Adapting word2vec to named entity recognition. In NCCL, pages 239-243, 2015.","title":"Scharolta Katharina Sienčnik. Adapting word2vec to named entity recognition","authors":"Sienčnik","year":"2015","venue":"NCCL","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Silver et al., 2013] Daniel L Silver, Qiang Yang, and Liang- hao Li. Lifelong Machine Learning Systems: Beyond Learning Algorithms. In AAAI Spring Symposium: LML, page 05, 2013.","title":"Lifelong Machine Learning Systems: Beyond Learning Algorithms","authors":"L Daniel, Qiang Silver, Lianghao Yang, Li","year":"2013","venue":"AAAI Spring Symposium: LML","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Thrun and Pratt, 2012] Sebastian Thrun and Lorien Pratt. Learning to learn. Springer, 2012.","title":"Sebastian Thrun and Lorien Pratt","authors":"Pratt Thrun","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Thrun, 1996] Sebastian Thrun. Is learning the n-th thing any easier than learning the first? In NIPS, pages 640-646, 1996.","title":"Is learning the n-th thing any easier than learning the first?","authors":"Sebastian Thrun, Thrun","year":"1996","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Xu et al., 2018a] Hu Xu, Bing Liu, Lei Shu, and Philip S. Yu. Double embeddings and cnn-based sequence labeling for aspect extraction. In ACL, pages 148-154, 2018.","title":"Double embeddings and cnn-based sequence labeling for aspect extraction","authors":"Xu","year":"2018","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Xu et al., 2018b] Hu Xu, Sihong Xie, Lei Shu, and Philip S. Yu. Dual attention network for product compatibility and function satisfiability analysis. In AAAI, pages 6013-6020, 2018.","title":"Dual attention network for product compatibility and function satisfiability analysis","authors":"Xu","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Yang et al., 2017] Wei Yang, Wei Lu, and Vincent Zheng. A simple regularization-based algorithm for learning cross- domain word embeddings. In EMNLP, pages 2898-2904, 2017.","title":"A simple regularization-based algorithm for learning crossdomain word embeddings","authors":"Wei Yang, Wei Lu, Vincent Zheng","year":"2017","venue":"EMNLP","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}