{"type":"ScholarlyArticle","title":"Unsupervised Multilingual Alignment using Wasserstein Barycenter","authors":[{"name":"Xin Lian","slug":"xin-lian","given":"Xin","family":"Lian","affiliation":"University of Waterloo , Waterloo , Canada; Vector Institute , Toronto , Canada"},{"name":"Kshitij Jain","slug":"kshitij-jain","given":"Kshitij","family":"Jain","affiliation":"Borealis AI , Waterloo , Canada"},{"name":"Jakub Truszkowski","slug":"jakub-truszkowski","given":"Jakub","family":"Truszkowski","affiliation":"Borealis AI , Waterloo , Canada"},{"name":"Pascal Poupart","slug":"pascal-poupart","given":"Pascal","family":"Poupart","affiliation":"University of Waterloo , Waterloo , Canada; Borealis AI , Waterloo , Canada; Vector Institute , Toronto , Canada"},{"name":"Yaoliang Yu","slug":"yaoliang-yu","given":"Yaoliang","family":"Yu","affiliation":"University of Waterloo , Waterloo , Canada; Vector Institute , Toronto , Canada"}],"abstract":"We study unsupervised multilingual alignment, the problem of finding word-to-word translations between multiple languages without using any parallel data. One popular strategy is to reduce multilingual alignment to the much simplified bilingual setting, by picking one of the input languages as the pivot language that we transit through. However, it is well-known that transiting through a poorly chosen pivot language (such as English) may severely degrade the translation quality, since the assumed transitive relations among all pairs of languages may not be enforced in the training process. Instead of going through a rather arbitrarily chosen pivot language, we propose to use the Wasserstein barycenter as a more informative ``mean'' language: it encapsulates information from all languages and minimizes all pairwise transportation costs. We evaluate our method on standard benchmarks and demonstrate state-of-the-art performances.","doi":"10.24963/ijcai.2020/512","url":"https://ijcai.org/proceedings/2020/512","pdf":"https://ijcai.org/proceedings/2020/0512.pdf","markdown":"https://ijcai.org/proceedings/2020/512.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/512","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Natural Language Processing","track":"Main track","number":512,"pages":{"first":3702,"last":3708},"keywords":["Machine Learning: Unsupervised Learning","Machine Learning Applications: Applications of Unsupervised Learning","Natural Language Processing: Machine Translation"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24908/file_24908.m3u8","length_sec":276,"provider":"hls"}],"references":[{"position":1,"raw":"[Agueh and Carlier, 2011] Martial Agueh and Guillaume Car- lier. Barycenters in the Wasserstein space. SIAM Journal on Mathematical Analysis, 43(2):904-924, 2011.","title":"Martial Agueh and Guillaume Carlier. Barycenters in the Wasserstein space","authors":"Carlier Agueh","year":"2011","venue":"SIAM Journal on Mathematical Analysis","doi":"10.1137/100805741","url":null,"internal_url":null},{"position":2,"raw":"[Alaux et al., 2019] Jean Alaux, Edouard Grave, Marco Cu- turi, and Armand Joulin. Unsupervised Hyper-alignment for Multilingual Word Embeddings. In ICLR, 2019.","title":"Unsupervised Hyper-alignment for Multilingual Word Embeddings","authors":"Jean Alaux, Edouard Grave, Marco Cuturi, Armand Joulin","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Alvarez-Melis and Jaakkola, 2018] David Alvarez-Melis and Tommi S Jaakkola. Gromov-Wasserstein Alignment of Word Embedding Spaces. In ACL, 2018.","title":"Gromov-Wasserstein Alignment of Word Embedding Spaces","authors":"Alvarez-Melis, David Jaakkola, Tommi S Alvarez-Melis, Jaakkola","year":"2018","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Artetxe et al., 2017] Mikel Artetxe, Gorka Labaka, and Eneko Agirre. Learning bilingual word embeddings with (almost) no bilingual data. In ACL, 2017.","title":"Learning bilingual word embeddings with (almost) no bilingual data","authors":"Mikel Artetxe, Gorka Labaka, Eneko Agirre","year":"2017","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Artetxe et al., 2018a] Mikel Artetxe, Gorka Labaka, and Eneko Agirre. Generalizing and Improving Bilingual Word Embedding Mappings with a Multi-Step Framework of Linear Transformations. In AAAI, 2018.","title":"Generalizing and Improving Bilingual Word Embedding Mappings with a Multi-Step Framework of Linear Transformations","authors":"Artetxe","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Artetxe et al., 2018b] Mikel Artetxe, Gorka Labaka, and Eneko Agirre. A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings. In ACL, 2018.","title":"A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings","authors":"Artetxe","year":"2018","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chen and Cardie, 2018] Xilun Chen and Claire Cardie. Un- supervised Multilingual Word Embeddings. EMNLP, 2018.","title":"Unsupervised Multilingual Word Embeddings","authors":"Chen, Cardie ;, Xilun Chen, Claire Cardie","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Claici et al., 2018] Sebastian Claici, Edward Chien, and Justin Solomon. Stochastic Wasserstein Barycenters. In ICML, 2018.","title":"Stochastic Wasserstein Barycenters","authors":"Sebastian Claici, Edward Chien, Justin Solomon","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Cuturi and Doucet, 2014] Marco Cuturi and Arnaud Doucet. Fast computation of wasserstein barycenters. ICML, 2014.","title":"Fast computation of wasserstein barycenters","authors":"Marco Cuturi, Arnaud Doucet","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Dinu et al., 2015] Georgiana Dinu, Angeliki Lazaridou, and Marco Baroni. Improving zero-shot learning by mitigating the hubness problem. In ICLR workshop, 2015.","title":"Improving zero-shot learning by mitigating the hubness problem","authors":"Georgiana Dinu, Angeliki Lazaridou, Marco Baroni","year":"2015","venue":"ICLR workshop","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Dou et al., 2018] Zi-Yi Dou, Zhi-Hao Zhou, and Shujian Huang. Unsupervised Bilingual Lexicon Induction via Latent Variable Models. In EMNLP, 2018.","title":"Unsupervised Bilingual Lexicon Induction via Latent Variable Models","authors":"Zi-Yi Dou, Zhi-Hao Zhou, Shujian Huang","year":"2018","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Fung, 1995] Pascale Fung. Compiling bilingual lexicon en- tries from a non-parallel english-chinese corpus. In Third Workshop on Very Large Corpora, 1995.","title":"Compiling bilingual lexicon entries from a non-parallel english-chinese corpus","authors":"Pascale Fung","year":"1995","venue":"Third Workshop on Very Large Corpora","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Gangbo and Świkech, 1998] Wilfrid Gangbo and Andrzej Świkech. Optimal maps for the multidimensional Monge- Kantorovich problem. Communications on Pure and Ap- plied Mathematics, 51(1):23-45, 1998.","title":"Wilfrid Gangbo and Andrzej Świkech. Optimal maps for the multidimensional Monge-Kantorovich problem","authors":"Świkech Gangbo","year":"1998","venue":"Communications on Pure and Applied Mathematics","doi":"10.1002/(sici)1097-0312(199801)51:1<23::aid-cpa2>3.0.co;2-h","url":null,"internal_url":null},{"position":14,"raw":"[Glavas et al., 2019] Goran Glavas, Robert Litschko, Sebas- tian Ruder, and Ivan Vulic. How to (Properly) Evaluate Cross-Lingual Word Embeddings: On Strong Baselines, Comparative Analyses, and Some Misconception. In ACL, 2019.","title":"How to (Properly) Evaluate Cross-Lingual Word Embeddings: On Strong Baselines, Comparative Analyses, and Some Misconception","authors":"Goran Glavas, Robert Litschko, Sebastian Ruder, Ivan Vulic","year":"2019","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Gouws et al., 2015] Stephan Gouws, Yoshua Bengio, and Greg Corrado. BilBOWA: Fast Bilingual Distributed Rep- resentations without Word Alignments. In ICML, 2015.","title":"BilBOWA: Fast Bilingual Distributed Representations without Word Alignments","authors":"Stephan Gouws, Yoshua Bengio, Greg Corrado","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Grave et al., 2019] Edouard Grave, Armand Joulin, and Quentin Berthet. Unsupervised Alignment of Embeddings with Wasserstein Procrustes. In AISTATS, 2019.","title":"Unsupervised Alignment of Embeddings with Wasserstein Procrustes","authors":"Grave","year":"2019","venue":"AISTATS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Hoshen and Wolf, 2018] Yedid Hoshen and Lior Wolf. Non- Adversarial Unsupervised Word Translation. In EMNLP, 2018.","title":"Yedid Hoshen and Lior Wolf. Non-Adversarial Unsupervised Word Translation","authors":"Wolf Hoshen","year":"2018","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Joulin et al., 2018] Armand Joulin, Piotr Bojanowski, Tomas Mikolov, Hervé Jégou, and Edouard Grave. Loss in Translation: Learning Bilingual Word Mapping with a Retrieval Criterion. In EMNLP, 2018.","title":"Loss in Translation: Learning Bilingual Word Mapping with a Retrieval Criterion","authors":"Armand Joulin, Piotr Bojanowski, Tomas Mikolov, Hervé Jégou, Edouard Grave","year":"2018","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lample et al., 2018] Guillaume Lample, Alexis Conneau, Marc'Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. Word translation without parallel data. In ICLR, 2018.","title":"Word translation without parallel data","authors":"Guillaume Lample, Alexis Conneau, Marc'aurelio Ranzato, Ludovic Denoyer, Hervé Jégou","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Mikolov et al., 2013a] Tomas Mikolov, Quoc V Le, and Ilya Sutskever. Exploiting Similarities among Languages for Machine Translation. CoRR, abs/1309.4168, 2013.","title":"Exploiting Similarities among Languages for Machine Translation","authors":"Mikolov","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Mikolov et al., 2013b] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed Repre- sentations of Words and Phrases and their Compositionality. In NIPS, 2013.","title":"Distributed Representations of Words and Phrases and their Compositionality","authors":"Mikolov","year":"2013","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Nakashole and Flauger, 2017] Ndapandula Nakashole and Raphael Flauger. Knowledge Distillation for Bilingual Dictionary Induction. In EMNLP, 2017.","title":"Ndapandula Nakashole and Raphael Flauger. Knowledge Distillation for Bilingual Dictionary Induction","authors":"Flauger Nakashole","year":"2017","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Rapp, 1995] Reinhard Rapp. Identifying Word Translations in Non-parallel Texts. In ACL, 1995.","title":"Identifying Word Translations in Non-parallel Texts","authors":"Reinhard Rapp, Rapp","year":"1995","venue":"ACL","doi":"10.3115/981658.981709","url":null,"internal_url":null},{"position":24,"raw":"[Schönemann, 1966] Peter H Schönemann. A generalized solution of the orthogonal procrustes problem. Psychome- trika, 31(1):1-10, Mar 1966.","title":"A generalized solution of the orthogonal procrustes problem","authors":"Peter H Schönemann","year":"1966","venue":"Psychometrika","doi":"10.1007/bf02289451","url":null,"internal_url":null},{"position":25,"raw":"[Smith et al., 2017] Samuel L Smith, David HP Turban, Steven Hamblin, and Nils Y Hammerla. Offline bilingual word vectors, orthogonal transformations and the inverted softmax. In ICLR, 2017.","title":"Offline bilingual word vectors, orthogonal transformations and the inverted softmax","authors":"L Samuel, David Smith, Steven Hp Turban, Nils Y Hamblin, Hammerla","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Taitelbaum et al., 2019a] Hagai Taitelbaum, Gal Chechik, and Jacob Goldberger. Multilingual word translation using auxiliary languages. In EMNLP, 2019.","title":"Multilingual word translation using auxiliary languages","authors":"Taitelbaum","year":"2019","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Taitelbaum et al., 2019b] Hagai Taitelbaum, Gal Chechik, and Jacob Goldberger. A multi-pairwise extension of procrustes analysis for multilingual word translation. In EMNLP, 2019.","title":"A multi-pairwise extension of procrustes analysis for multilingual word translation","authors":"Taitelbaum","year":"2019","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wada et al., 2019] Takashi Wada, Tomoharu Iwata, and Yuji Matsumoto. Unsupervised Multilingual Word Embedding with Limited Resources using Neural Language Models. In ACL, 2019.","title":"Unsupervised Multilingual Word Embedding with Limited Resources using Neural Language Models","authors":"Takashi Wada, Tomoharu Iwata, Yuji Matsumoto","year":"2019","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Xing et al., 2015] Chao Xing, Dong Wang, Chao Liu, and Yiye Lin. Normalized word embedding and orthogonal transform for bilingual word translation. In NAACL, 2015.","title":"Normalized word embedding and orthogonal transform for bilingual word translation","authors":"Chao Xing, Dong Wang, Chao Liu, Yiye Lin","year":"2015","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhang et al., 2016] Yuan Zhang, David Gaddy, Regina Barzilay, and Tommi Jaakkola. Ten pairs to tag-multilingual pos tagging via coarse mapping between embeddings. In ACL, 2016.","title":"Ten pairs to tag-multilingual pos tagging via coarse mapping between embeddings","authors":"Yuan Zhang, David Gaddy, Regina Barzilay, Tommi Jaakkola","year":"2016","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhang et al., 2017a] Meng Zhang, Yang Liu, Huanbo Luan, and Maosong Sun. Earth Mover's Distance Minimization for Unsupervised Bilingual Lexicon Induction. In EMNLP, 2017.","title":"Earth Mover's Distance Minimization for Unsupervised Bilingual Lexicon Induction","authors":"Zhang","year":"2017","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhang et al., 2017b] Meng Zhang, Yang Liu, Huanbo Luan, and Maosong Sun. Adversarial Training for Unsupervised Bilingual Lexicon Induction. In ACL, 2017.","title":"Adversarial Training for Unsupervised Bilingual Lexicon Induction","authors":"Zhang","year":"2017","venue":"ACL","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}