{"type":"ScholarlyArticle","title":"A Variational Autoencoding Approach for Inducing Cross-lingual Word Embeddings","authors":[{"name":"Liangchen Wei","slug":"liangchen-wei","given":"Liangchen","family":"Wei","affiliation":"Key Laboratory of Machine Perception (Ministry of Education) , School of Electronics Engineering and Computer Science , Peking University , Beijing 100871 , China"},{"name":"Zhi-Hong Deng","slug":"zhi-hong-deng","given":"Zhi-Hong","family":"Deng","affiliation":"Key Laboratory of Machine Perception (Ministry of Education) , School of Electronics Engineering and Computer Science , Peking University , Beijing 100871 , China"}],"abstract":"Cross-language learning allows one to use training data from one language to build models for another language. Many traditional approaches require word-level alignment sentences from parallel corpora, in this paper we define a general bilingual training objective function requiring sentence level parallel corpus only. We propose a variational autoencoding approach for training bilingual word embeddings. The variational model introduces a continuous latent variable to explicitly model the underlying semantics of the parallel sentence pairs and to guide the generation of the sentence pairs. Our model restricts the bilingual word embeddings to represent words in exactly the same continuous vector space. Empirical results on the task of cross lingual document classification has shown that our method is effective.","doi":"10.24963/ijcai.2017/582","url":"https://ijcai.org/proceedings/2017/582","pdf":"https://ijcai.org/proceedings/2017/0582.pdf","markdown":"https://ijcai.org/proceedings/2017/582.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/582","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Natural Language Processing","track":"Main track","number":582,"pages":{"first":4165,"last":4171},"keywords":["Natural Language Processing: Text Classification","Natural Language Processing: Natural Language Semantics"],"videos":[],"references":[{"position":1,"raw":"[Abadi et al., 2016] Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, et al. Tensorflow: Large-scale machine learn- ing on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467, 2016.","title":"Tensorflow: Large-scale machine learning on heterogeneous distributed systems","authors":"Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[AP et al., 2014] Sarath Chandar AP, Stanislas Lauly, Hugo Larochelle, Mitesh Khapra, Balaraman Ravindran, Vikas C Raykar, and Amrita Saha. An autoencoder approach to learning bilingual word representations. In Advances in Neural Information Processing Systems, pages 1853-1861, 2014.","title":"An autoencoder approach to learning bilingual word representations","authors":"Sarath Chandar, A P, Stanislas Lauly, Hugo Larochelle, Mitesh Khapra, Balaraman Ravindran, Amrita Vikas C Raykar, Saha","year":"2014","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bengio et al., 2003] Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. A neural probabilis- tic language model. Journal of machine learning research, 3(Feb):1137-1155, 2003.","title":"A neural probabilistic language model","authors":"Yoshua Bengio, Réjean Ducharme, Pascal Vincent, Christian Jauvin","year":"2003","venue":"Journal of machine learning research","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bowman et al., 2015] Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Bengio. Generating sentences from a continuous space. arXiv preprint arXiv:1511.06349, 2015.","title":"Generating sentences from a continuous space","authors":"Luke Samuel R Bowman, Oriol Vilnis, Andrew M Vinyals, Rafal Dai, Samy Jozefowicz, Bengio","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Faruqui and Dyer, 2014] Manaal Faruqui and Chris Dyer. Improving vector space word representations using mul- tilingual correlation. Association for Computational Lin- guistics, 2014.","title":"Improving vector space word representations using multilingual correlation","authors":"Dyer Faruqui, Manaal Faruqui, Chris Dyer","year":"2014","venue":"Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Freund and Schapire, 1999] Yoav Freund and Robert E Schapire. Large margin classification using the perceptron algorithm. Machine learning, 37(3):277-296, 1999.","title":"Large margin classification using the perceptron algorithm","authors":"Schapire Freund, Yoav Freund, Robert E Schapire","year":"1999","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"Gouws et al., ] S Gouws, Y Bengio, and G Corrado. Bil- bowa: fast bilingual distributed representations without word alignments (2014). arXiv preprint arXiv:1410.2455.","title":"Bilbowa: fast bilingual distributed representations without word alignments","authors":"Gouws","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hermann and Blunsom, 2013] Karl Moritz Hermann and Phil Blunsom. Multilingual distributed representations without word alignment. arXiv preprint arXiv:1312.6173, 2013.","title":"Multilingual distributed representations without word alignment","authors":"Blunsom Hermann, Karl Moritz Hermann, Phil Blunsom","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ioffe and Szegedy, 2015] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep net- work training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.","title":"Batch normalization: Accelerating deep network training by reducing internal covariate shift","authors":"Szegedy Ioffe, Sergey Ioffe, Christian Szegedy","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kingma and Ba, 2014] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"Diederik Kingma, Jimmy Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kingma and Welling, 2013] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.","title":"Auto-encoding variational bayes","authors":"P Diederik, Max Kingma, Welling","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Klementiev et al., 2012] Alexandre Klementiev, Ivan Titov, and Binod Bhattarai. Inducing crosslingual distributed representations of words. 2012.","title":"Inducing crosslingual distributed representations of words","authors":"Alexandre Klementiev, Ivan Titov, Binod Bhattarai","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Koehn, 2005] Philipp Koehn. Europarl: A parallel corpus for statistical machine translation. In MT summit, vol- ume 5, pages 79-86, 2005.","title":"Europarl: A parallel corpus for statistical machine translation","authors":"Philipp Koehn, Koehn","year":"2005","venue":"MT summit","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lewis et al., 2004] David D Lewis, Yiming Yang, Tony G Rose, and Fan Li. Rcv1: A new benchmark collection for text categorization research. Journal of machine learning research, 5(Apr):361-397, 2004.","title":"Rcv1: A new benchmark collection for text categorization research","authors":"Yiming David D Lewis, Tony G Yang, Fan Rose, Li","year":"2004","venue":"Journal of machine learning research","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Luong et al., 2015] Thang Luong, Hieu Pham, and Christo- pher D Manning. Bilingual word representations with monolingual quality in mind. In Proceedings of the 1st Workshop on Vector Space Modeling for Natural Lan- guage Processing, pages 151-159, 2015.","title":"Bilingual word representations with monolingual quality in mind","authors":"Thang Luong, Hieu Pham, Christopher D Manning","year":"2015","venue":"Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Maaten and Hinton, 2008] Laurens van der Maaten and Ge- offrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9(Nov):2579-2605, 2008.","title":"Laurens van der Maaten and Geoffrey Hinton","authors":"Hinton Maaten","year":"2008","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Miao et al., 2016] Yishu Miao, Lei Yu, and Phil Blunsom. Neural variational inference for text processing. In Proc. ICML, 2016.","title":"Neural variational inference for text processing","authors":"Yishu Miao, Lei Yu, Phil Blunsom","year":"2016","venue":"Proc. ICML","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mikolov et al., 2013] Tomas Mikolov, Quoc V Le, and Ilya Sutskever. Exploiting similarities among languages for machine translation. arXiv preprint arXiv:1309.4168, 2013.","title":"and Ilya Sutskever. Exploiting similarities among languages for machine translation","authors":"Mikolov","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Rezende et al., 2014] Danilo Jimenez Rezende, Shakir Mo- hamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. arXiv preprint arXiv:1401.4082, 2014.","title":"Stochastic backpropagation and approximate inference in deep generative models","authors":"Danilo Jimenez Rezende, Shakir Mohamed, Daan Wierstra","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Shi et al., 2015] Tianze Shi, Zhiyuan Liu, Yang Liu, and Maosong Sun. Learning cross-lingual word embeddings via matrix co-factorization. In ACL (2), pages 567-572, 2015.","title":"Learning cross-lingual word embeddings via matrix co-factorization","authors":"Shi","year":"2015","venue":"ACL (2)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Socher et al., 2011] Richard Socher, Jeffrey Pennington, Eric H Huang, Andrew Y Ng, and Christopher D Man- ning. Semi-supervised recursive autoencoders for predict- ing sentiment distributions. In Proceedings of the confer- ence on empirical methods in natural language process- ing, pages 151-161. Association for Computational Lin- guistics, 2011.","title":"Semi-supervised recursive autoencoders for predicting sentiment distributions","authors":"Richard Socher, Jeffrey Pennington, Eric H Huang, Andrew Y Ng, Christopher D Manning","year":"2011","venue":"Proceedings of the conference on empirical methods in natural language processing","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Srivastava et al., 2014] Nitish Srivastava, Geoffrey E Hin- ton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929-1958, 2014.","title":"Dropout: a simple way to prevent neural networks from overfitting","authors":"Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov","year":"2014","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Šuster et al., 2016] Simon Šuster, Ivan Titov, and Gert- jan van Noord. Bilingual learning of multi-sense em- beddings with discrete autoencoders. arXiv preprint arXiv:1603.09128, 2016.","title":"Bilingual learning of multi-sense embeddings with discrete autoencoders","authors":"Simon Šuster, Ivan Titov, Gertjan Van Noord","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Suzuki et al., 2016] Masahiro Suzuki, Kotaro Nakayama, and Yutaka Matsuo. Joint multimodal learning with deep generative models. arXiv preprint arXiv:1611.01891, 2016.","title":"Joint multimodal learning with deep generative models","authors":"Masahiro Suzuki, Kotaro Nakayama, Yutaka Matsuo","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Täckström et al., 2012] Oscar Täckström, Ryan McDonald, and Jakob Uszkoreit. Cross-lingual word clusters for di- rect transfer of linguistic structure. In Proceedings of the 2012 conference of the North American chapter of the as- sociation for computational linguistics: Human language technologies, pages 477-487. Association for Computa- tional Linguistics, 2012.","title":"Cross-lingual word clusters for direct transfer of linguistic structure","authors":"Täckström","year":"2012","venue":"Proceedings of the 2012 conference of the North American chapter of the association for computational linguistics: Human language technologies","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Vulic and Moens, 2013a] Ivan Vulic and Marie-Francine Moens. Cross-lingual semantic similarity of words as the similarity of their semantic word responses. In Proceed- ings of the 2013 Conference of the North American Chap- ter of the Association for Computational Linguistics: Hu- man Language Technologies (NAACL-HLT 2013), pages 106-116. ACL, 2013.","title":"Cross-lingual semantic similarity of words as the similarity of their semantic word responses","authors":"Moens Vulic, Ivan Vulic, Marie-Francine Moens","year":"2013","venue":"Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT 2013)","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Vulic and Moens, 2013b] Ivan Vulic and Marie-Francine Moens. A study on bootstrapping bilingual vector spaces from non-parallel data (and nothing else). In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (EMNLP 2013), pages 1613-1624. ACL, 2013.","title":"Ivan Vulic and Marie-Francine Moens. A study on bootstrapping bilingual vector spaces from non-parallel data (and nothing else)","authors":"Moens Vulic","year":"2013","venue":"Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (EMNLP 2013)","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhang et al., 2016] Biao Zhang, Deyi Xiong, Jinsong Su, Hong Duan, and Min Zhang. Variational neural machine translation. arXiv preprint arXiv:1605.07869, 2016.","title":"Variational neural machine translation","authors":"Biao Zhang, Deyi Xiong, Jinsong Su, Hong Duan, Min Zhang","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zou et al., 2013] Will Y Zou, Richard Socher, Daniel M Cer, and Christopher D Manning. Bilingual word embed- dings for phrase-based machine translation. In EMNLP, pages 1393-1398, 2013.","title":"Bilingual word embeddings for phrase-based machine translation","authors":"Will Y Zou, Richard Socher, Daniel M Cer, Christopher D Manning","year":"2013","venue":"EMNLP","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}