{"type":"ScholarlyArticle","title":"Joint Learning Embeddings for Chinese Words and their Components via Ladder Structured Networks","authors":[{"name":"Yan Song","slug":"yan-song","given":"Yan","family":"Song","affiliation":"Tencent AI Lab"},{"name":"Shuming Shi","slug":"shuming-shi","given":"Shuming","family":"Shi","affiliation":"Tencent AI Lab"},{"name":"Jing Li","slug":"jing-li","given":"Jing","family":"Li","affiliation":"Tencent AI Lab"}],"abstract":"The components, such as characters and radicals, of a Chinese word are important sources to help in capturing semantic information of the word. In this paper, we propose a novel framework, namely, ladder structured networks (LSN), which contains three layers representing word, character and radical and learns their embeddings synchronously. LSN captures not only the relations among words, but also the relations among their component characters and radicals, as well as the relations across layers. Each layer in LSN is pluggable so that any particular type of unit (word, character, radical) can be removed and the LSN is thus adjusted for particular types of inputs. In evaluating our framework, we use word similarity as the intrinsic evaluation and part-of-speech tagging and document classification as extrinsic evaluations. Experimental results confirm the validity of our approach and show superiority of our approach over previous work.","doi":"10.24963/ijcai.2018/608","url":"https://ijcai.org/proceedings/2018/608","pdf":"https://ijcai.org/proceedings/2018/0608.pdf","markdown":"https://ijcai.org/proceedings/2018/608.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/608","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Natural Language Processing","track":"Main track","number":608,"pages":{"first":4375,"last":4381},"keywords":["Natural Language Processing: Embeddings","Natural Language Processing: Natural Language Processing"],"videos":[],"references":[{"position":1,"raw":"[Bengio et al., 2003] Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. A Neural Proba- bilistic Language Model. Journal of Machine Learning Research, 3:1137-1155, March 2003.","title":"A Neural Probabilistic Language Model","authors":"Yoshua Bengio, Réjean Ducharme, Pascal Vincent, Christian Janvin","year":"2003","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Botha and Blunsom, 2014] Jan A. Botha and Phil Blunsom. Compositional Morphology for Word Representations and Language Modelling. arXiv preprint, abs/1405.4273, 2014.","title":"Compositional Morphology for Word Representations and Language Modelling","authors":"Jan A Botha, Phil Blunsom","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2015] Xinxiong Chen, Lei Xu, Zhiyuan Liu, Maosong Sun, and Huanbo Luan. Joint Learning of Char- acter and Word Embeddings. In Proceedings of IJCAI, pages 1236-1242, 2015.","title":"Joint Learning of Character and Word Embeddings","authors":"Chen","year":"2015","venue":"Proceedings of IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2015/178"},{"position":4,"raw":"[Collobert et al., 2011] Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. Natural Language Processing (Almost) from Scratch. Journal of Machine Learning Research, 12:2493- 2537, Nov 2011.","title":"Natural Language Processing (Almost) from Scratch","authors":"Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, Pavel Kuksa","year":"2011","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Huang et al., 2015] Zhiheng Huang, Wei Xu, and Kai Yu. Bidirectional LSTM-CRF Models for Sequence Tagging. arXiv preprint, abs/1508.01991, 2015.","title":"Bidirectional LSTM-CRF Models for Sequence Tagging","authors":"Zhiheng Huang, Wei Xu, Kai Yu","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Jégou et al., 2016] Simon Jégou, Michal Drozdzal, David Vázquez, Adriana Romero, and Yoshua Bengio. The One Hundred Layers Tiramisu: Fully Convolutional DenseNets for Semantic Segmentation. arXiv preprint, abs/1611.09326, 2016.","title":"The One Hundred Layers Tiramisu: Fully Convolutional DenseNets for Semantic Segmentation","authors":"Simon Jégou, Michal Drozdzal, David Vázquez, Adriana Romero, Yoshua Bengio","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Jin and Wu, 2012] Peng Jin and Yunfang Wu. SemEval- 2012 Task 4: Evaluating Chinese Word Similarity. In *SEM, pages 374-377, Montréal, Canada, 7-8 June 2012.","title":"SemEval-2012 Task 4: Evaluating Chinese Word Similarity","authors":"Jin Peng, Yunfang Wu","year":"2012","venue":"SEM","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kiela et al., 2015] Douwe Kiela, Felix Hill, and Stephen Clark. Specializing Word Embeddings for Similarity or Relatedness. In Proceedings of EMNLP, pages 2044- 2048, Lisbon, Portugal, September 2015.","title":"Specializing Word Embeddings for Similarity or Relatedness","authors":"Kiela","year":"2015","venue":"Proceedings of EMNLP","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kiros et al., 2015] Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S. Zemel, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. Skip-thought Vectors. In NIPS, pages 3294-3302, Cambridge, MA, USA, 2015.","title":"Skip-thought Vectors","authors":"Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S Zemel, Antonio Torralba, Raquel Urtasun, Sanja Fidler","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Le and Mikolov, 2014] Quoc V. Le and Tomas Mikolov. Distributed Representations of Sentences and Documents. In Proceedings of ICML, pages 1188-1196, Beijing, China, June 2014.","title":"Distributed Representations of Sentences and Documents","authors":"V Quoc, Tomas Le, Mikolov","year":"2014","venue":"Proceedings of ICML","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2015] Yanran Li, Wenjie Li, Fei Sun, and Su- jian Li. Component-Enhanced Chinese Character Embed- dings. In Proceedings of EMNLP, pages 829-834, Lisbon, Portugal, September 2015.","title":"Component-Enhanced Chinese Character Embeddings","authors":"Yanran Li, Wenjie Li, Fei Sun, Sujian Li","year":"2015","venue":"Proceedings of EMNLP","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Luong et al., 2013] Thang Luong, Richard Socher, and Christopher Manning. Better Word Representations with Recursive Neural Networks for Morphology. In Proceed- ings of CoNLL, pages 104-113, Sofia, Bulgaria, 2013.","title":"Better Word Representations with Recursive Neural Networks for Morphology","authors":"Thang Luong, Richard Socher, Christopher Manning","year":"2013","venue":"Proceedings of CoNLL","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mikolov et al., 2013a] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient Estimation of Word Representations in Vector Space. arXiv preprint, abs/1301.3781, 2013.","title":"Efficient Estimation of Word Representations in Vector Space","authors":"Mikolov","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mikolov et al., 2013b] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed Repre- sentations of Words and Phrases and their Compositional- ity. In NIPS, pages 3111-3119. 2013.","title":"Distributed Representations of Words and Phrases and their Compositionality","authors":"Mikolov","year":"2013","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Miyamoto and Cho, 2016] Yasumasa Miyamoto and Kyunghyun Cho. Gated Word-Character Recurrent Language Model. arXiv preprint, abs/1606.01700, 2016.","title":"Gated Word-Character Recurrent Language Model","authors":"Cho Miyamoto, Yasumasa Miyamoto, Kyunghyun Cho","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Pennington et al., 2014] Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global Vectors for Word Representation. In Proceedings of EMNLP, pages 1532-1543, Doha, Qatar, October 2014.","title":"Glove: Global Vectors for Word Representation","authors":"Jeffrey Pennington, Richard Socher, Christopher Manning","year":"2014","venue":"Proceedings of EMNLP","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Rasmus et al., 2015a] Antti Rasmus, Harri Valpola, Mikko Honkala, Mathias Berglund, and Tapani Raiko. Semi- Supervised Learning with Ladder Network. arXiv preprint, abs/1507.02672, 2015.","title":"Semi-Supervised Learning with Ladder Network","authors":"Rasmus","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Rasmus et al., 2015b] Antti Rasmus, Harri Valpola, and Tapani Raiko. Lateral Connections in Denoising Au- toencoders Support Supervised Learning. arXiv preprint, abs/1504.08215, 2015.","title":"Lateral Connections in Denoising Autoencoders Support Supervised Learning","authors":"Rasmus","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ronneberger et al., 2015] Olaf Ronneberger, Philipp Fis- cher, and Thomas Brox. U-Net: Convolutional Networks for Biomedical Image Segmentation. arXiv preprint, abs/1505.04597, 2015.","title":"Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional Networks for Biomedical Image Segmentation","authors":"Ronneberger","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Su and Lee, 2017] Tzu-ray Su and Hung-yi Lee. Learning Chinese Word Representations From Glyphs Of Charac- ters. In Proceedings of EMNLP, pages 264-273, Copen- hagen, Denmark, September 2017.","title":"Tzu-ray Su and Hung-yi Lee. Learning Chinese Word Representations From Glyphs Of Characters","authors":"Lee Su","year":"2017","venue":"Proceedings of EMNLP","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Trask et al., 2015] Andrew Trask, David Gilmore, and Matthew Russell. Modeling Order in Neural Word Em- beddings at Scale. arXiv pre-print, abs/1506.02338, 2015.","title":"Modeling Order in Neural Word Embeddings at Scale","authors":"Andrew Trask, David Gilmore, Matthew Russell","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Valpola, 2014] Harri Valpola. From Neural PCA to Deep Unsupervised Learning. arXiv pre-print, abs/1411.7783, 2014.","title":"From Neural PCA to Deep Unsupervised Learning","authors":"Harri Valpola","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[van der Maaten and Hinton, 2008] L.J.P. van der Maaten and G.E. Hinton. Visualizing high-dimensional data using t-SNE. Journal of Machine Learning Research, 9:2579- 2605, 2008.","title":"Visualizing high-dimensional data using t-SNE","authors":"Van Der Maaten, ; ] L J P Hinton, G E Van Der Maaten, Hinton","year":"2008","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Xu et al., 2016] Jian Xu, Jiawei Liu, Liangang Zhang, Zhengyu Li, and Huanhuan Chen. Improve Chinese Word Embeddings by Exploiting Internal Structure. In Proceed- ings of NAACL-HLT, pages 1041-1050, San Diego, Cali- fornia, June 2016.","title":"Improve Chinese Word Embeddings by Exploiting Internal Structure","authors":"Jian Xu, Jiawei Liu, Liangang Zhang, Zhengyu Li, Huanhuan Chen","year":"2016","venue":"Proceedings of NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Xue et al., 2005] Naiwen Xue, Fei Xia, Fu-dong Chiou, and Marta Palmer. The Penn Chinese TreeBank: Phrase Struc- ture Annotation of a Large Corpus. Natural Language En- gineering, 11(2):207-238, 2005.","title":"The Penn Chinese TreeBank: Phrase Structure Annotation of a Large Corpus","authors":"Naiwen Xue, Fei Xia, Fu-Dong Chiou, Marta Palmer","year":"2005","venue":"Natural Language Engineering","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yin et al., 2016] Rongchao Yin, Quan Wang, Peng Li, Rui Li, and Bin Wang. Multi-Granularity Chinese Word Em- bedding. In Proceedings of EMNLP, pages 981-986, Austin, Texas, November 2016.","title":"Multi-Granularity Chinese Word Embedding","authors":"Rongchao Yin, Quan Wang, Peng Li, Rui Li, Bin Wang","year":"2016","venue":"Proceedings of EMNLP","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}