{"type":"ScholarlyArticle","title":"Improving Multilingual Sentence Embedding using Bi-directional Dual Encoder with Additive Margin Softmax","authors":[{"name":"Yinfei Yang","slug":"yinfei-yang","given":"Yinfei","family":"Yang","affiliation":""},{"name":"Gustavo Hernandez Abrego","slug":"gustavo-hernandez-abrego","given":"Gustavo","family":"Hernandez Abrego","affiliation":""},{"name":"Steve Yuan","slug":"steve-yuan","given":"Steve","family":"Yuan","affiliation":""},{"name":"Mandy Guo","slug":"mandy-guo","given":"Mandy","family":"Guo","affiliation":""},{"name":"Qinlan Shen","slug":"qinlan-shen","given":"Qinlan","family":"Shen","affiliation":""},{"name":"Daniel Cer","slug":"daniel-cer","given":"Daniel","family":"Cer","affiliation":""},{"name":"Yun-hsuan Sung","slug":"yun-hsuan-sung","given":"Yun-hsuan","family":"Sung","affiliation":""},{"name":"Brian Strope","slug":"brian-strope","given":"Brian","family":"Strope","affiliation":""},{"name":"Ray Kurzweil","slug":"ray-kurzweil","given":"Ray","family":"Kurzweil","affiliation":""}],"abstract":"In this paper, we present an approach to learn multilingual sentence embeddings using a bi-directional dual-encoder with additive margin softmax. The embeddings are able to achieve state-of-the-art results on the United Nations (UN) parallel corpus retrieval task. In all the languages tested, the system achieves P@1 of 86% or higher. We use pairs retrieved by our approach to train NMT models that achieve similar performance to models trained on gold pairs. We explore simple document-level embeddings constructed by averaging our sentence embeddings. On the UN document-level retrieval task, document embeddings achieve around 97% on P@1 for all experimented language pairs. Lastly, we evaluate the proposed model on the BUCC mining task. The learned embeddings with raw cosine similarity scores achieve competitive results compared to current state-of-the-art models, and with a second-stage scorer we achieve a new state-of-the-art level on this task.","doi":"10.24963/ijcai.2019/746","url":"https://ijcai.org/proceedings/2019/746","pdf":"https://ijcai.org/proceedings/2019/0746.pdf","markdown":"https://ijcai.org/proceedings/2019/746.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/746","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Natural Language Processing","track":"Main track","number":746,"pages":{"first":5370,"last":5378},"keywords":["Natural Language Processing: Information Retrieval","Natural Language Processing: Natural Language Processing","Natural Language Processing: Machine Translation","Natural Language Processing: Embeddings"],"videos":[],"references":[{"position":1,"raw":"[Antonova and Misyurev, 2011] Alexandra Antonova and Alexey Misyurev. Building a web-based parallel corpus and filtering out machine-translated text. In Proceedings of the 4th Workshop on Building and Using Comparable Corpora: Comparable Corpora and the Web, pages 136- 144. Association for Computational Linguistics, 2011.","title":"Building a web-based parallel corpus and filtering out machine-translated text","authors":"Misyurev Antonova, Alexandra Antonova, Alexey Misyurev","year":"2011","venue":"Proceedings of the 4th Workshop on Building and Using Comparable Corpora: Comparable Corpora and the Web","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Artetxe and Schwenk, 2018] Mikel Artetxe and Holger Schwenk. Margin-based parallel corpus mining with mul- tilingual sentence embeddings. CoRR, abs/1811.01136, 2018.","title":"Margin-based parallel corpus mining with multilingual sentence embeddings","authors":"Artetxe, Schwenk, Mikel Artetxe, Holger Schwenk","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bojar et al., 2013] Ondřej Bojar, Christian Buck, Chris Callison-Burch, Christian Federmann, Barry Haddow, Philipp Koehn, Christof Monz, Matt Post, Radu Soricut, and Lucia Specia. Findings of the 2013 Workshop on Statistical Machine Translation. In Proceedings of the Eighth Workshop on Statistical Machine Translation, Au- gust 2013.","title":"Findings of the 2013 Workshop on Statistical Machine Translation","authors":"Ondřej Bojar, Christian Buck, Chris Callison-Burch, Christian Federmann, Barry Haddow, Philipp Koehn, Christof Monz, Matt Post, Radu Soricut, Lucia Specia","year":"2013","venue":"Proceedings of the Eighth Workshop on Statistical Machine Translation","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bojar et al., 2014] Ondřej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Lev- eling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, et al. Findings of the 2014 workshop on statistical machine translation. In Proceedings of the ninth workshop on statistical machine translation, pages 12-58, 2014.","title":"Findings of the 2014 workshop on statistical machine translation","authors":"Ondřej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand","year":"2014","venue":"Proceedings of the ninth workshop on statistical machine translation","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bouamor and Sajjad, 2018] Houda Bouamor and Hassan Sajjad. H2@bucc18: Parallel sentence extraction from comparable corpora using multilingual sentence embed- dings. In Reinhard Rapp, Pierre Zweigenbaum, and Serge Sharoff, editors, Proceedings of the Eleventh Interna- tional Conference on Language Resources and Evaluation (LREC 2018), May 2018.","title":"H2@bucc18: Parallel sentence extraction from comparable corpora using multilingual sentence embeddings","authors":"Houda Bouamor, Hassan Sajjad","year":"2018","venue":"Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen and Nie, 2000] Jiang Chen and Jian-Yun Nie. Parallel web text mining for cross-language ir. In Content-Based Multimedia Information Access-Volume 1, pages 62-77. Centre de Hautes Etudes Internationale D'Informatique Documentaire, 2000.","title":"Parallel web text mining for cross-language ir","authors":"Jiang Chen, Jian-Yun Nie","year":"2000","venue":"Content-Based Multimedia Information Access","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chidambaram et al., 2018] Muthuraman Chidambaram, Yinfei Yang, Daniel Cer, Steve Yuan, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. Learning cross-lingual sentence representations via a multi-task dual-encoder model. CoRR, abs/1810.12836, 2018.","title":"Learning cross-lingual sentence representations via a multi-task dual-encoder model","authors":"Muthuraman Chidambaram, Yinfei Yang, Daniel Cer, Steve Yuan, Yun-Hsuan Sung, Brian Strope, Ray Kurzweil","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Devlin et al., 2018] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understand- ing. CoRR, abs/1810.04805, 2018.","title":"BERT: pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Do et al., 2009] Thi-Ngoc-Diep Do, Viet-Bac Le, Brigitte Bigi, Laurent Besacier, and Eric Castelli. Mining a com- parable text corpus for a vietnamese-french statistical ma- chine translation system. In Proceedings of the Fourth Workshop on Statistical Machine Translation, pages 165- 172. Association for Computational Linguistics, 2009.","title":"Mining a comparable text corpus for a vietnamese-french statistical machine translation system","authors":"Thi-Ngoc-Diep Do, Viet-Bac Le, Brigitte Bigi, Laurent Besacier, Eric Castelli","year":"2009","venue":"Proceedings of the Fourth Workshop on Statistical Machine Translation","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[dos Santos et al., 2016] Cícero Nogueira dos Santos, Ming Tan, Bing Xiang, and Bowen Zhou. Attentive pooling net- works. CoRR, abs/1602.03609, 2016.","title":"Attentive pooling networks","authors":"Cícero Nogueira Dos Santos, Ming Tan, Bing Xiang, Bowen Zhou","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Grégoire and Langlais, 2017] Francis Grégoire and Philippe Langlais. A deep neural network ap- proach to parallel sentence extraction. arXiv preprint arXiv:1709.09783, 2017.","title":"A deep neural network approach to parallel sentence extraction","authors":"Francis Grégoire, Philippe Langlais","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Guo et al., 2018] Mandy Guo, Qinlan Shen, Yinfei Yang, Heming Ge, Daniel Cer, Gustavo Hernandez Abrego, Keith Stevens, Noah Constant, Yun-hsuan Sung, Brian Strope, and Ray Kurzweil. Effective parallel corpus min- ing using bilingual sentence embeddings. In Proceed- ings of the Third Conference on Machine Translation: Re- search Papers, pages 165-176. Association for Computa- tional Linguistics, 2018.","title":"Effective parallel corpus mining using bilingual sentence embeddings","authors":"Mandy Guo, Qinlan Shen, Yinfei Yang, Heming Ge, Daniel Cer, Gustavo Hernandez Abrego, Keith Stevens, Noah Constant, Yun-Hsuan Sung, Brian Strope, Ray Kurzweil","year":"2018","venue":"Proceedings of the Third Conference on Machine Translation: Research Papers","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hassan et al., 2018] Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Fe- dermann, Xuedong Huang, Marcin Junczys-Dowmunt, William Lewis, Mu Li, et al. Achieving human parity on automatic chinese to english news translation. arXiv preprint arXiv:1803.05567, 2018.","title":"Achieving human parity on automatic chinese to english news translation","authors":"Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dowmunt, William Lewis, Mu Li","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Munteanu and Marcu, 2005] Dragos Stefan Munteanu and Daniel Marcu. Improving machine translation perfor- mance by exploiting non-parallel corpora. Computational Linguistics, 31(4):477-504, 2005.","title":"Improving machine translation performance by exploiting non-parallel corpora","authors":"Stefan Dragos, Daniel Munteanu, Marcu","year":"2005","venue":"Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Munteanu and Marcu, 2006] Dragos Stefan Munteanu and Daniel Marcu. Extracting parallel sub-sentential frag- ments from non-parallel corpora. In Proceedings of the 21st International Conference on Computational Linguis- tics and the 44th Annual Meeting of the Association for Computational Linguistics, pages 81-88. Association for Computational Linguistics, 2006.","title":"Extracting parallel sub-sentential fragments from non-parallel corpora","authors":"Stefan Dragos, Daniel Munteanu, Marcu","year":"2006","venue":"Proceedings of the 21st International Conference on Computational Linguistics and the 44th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Papineni et al., 2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for auto- matic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311-318. Association for Computational Linguistics, 2002.","title":"Bleu: a method for automatic evaluation of machine translation","authors":"Kishore Papineni, Salim Roukos, Todd Ward, Wei-Jing Zhu","year":"2002","venue":"Proceedings of the 40th annual meeting on association for computational linguistics","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Resnik and Smith, 2003] Philip Resnik and Noah A Smith. The web as a parallel corpus. Computational Linguistics, 29(3):349-380, 2003.","title":"The web as a parallel corpus","authors":"Resnik, Smith ;, Noah A Smith","year":"2003","venue":"Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Resnik, 1999] Philip Resnik. Mining the web for bilingual text. In Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics on Computa- tional Linguistics, pages 527-534. Association for Com- putational Linguistics, 1999.","title":"Mining the web for bilingual text","authors":"Philip Resnik","year":"1999","venue":"Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics on Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Schuster and Nakajima, 2012] M. Schuster and K. Naka- jima. Japanese and korean voice search. In 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5149-5152, March 2012.","title":"Japanese and korean voice search","authors":"M Schuster, K Nakajima","year":"2012","venue":"IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Schwenk, 2018] Holger Schwenk. Filtering and mining par- allel data in a joint multilingual space. arXiv preprint arXiv:1805.09822, 2018.","title":"Filtering and mining parallel data in a joint multilingual space","authors":"Holger Schwenk","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Shi et al., 2006] Lei Shi, Cheng Niu, Ming Zhou, and Jian- feng Gao. A dom tree alignment model for mining par- allel data from the web. In Proceedings of the 21st In- ternational Conference on Computational Linguistics and the 44th Annual Meeting of the Association for Computa- tional Linguistics, pages 489-496. Association for Com- putational Linguistics, 2006.","title":"A dom tree alignment model for mining parallel data from the web","authors":"Lei Shi, Cheng Niu, Ming Zhou, Jianfeng Gao","year":"2006","venue":"Proceedings of the 21st International Conference on Computational Linguistics and the 44th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Uszkoreit et al., 2010] Jakob Uszkoreit, Jay M. Ponte, Ashok C. Popat, and Moshe Dubiner. Large scale par- allel document mining for machine translation. In Pro- ceedings of the 23rd International Conference on Compu- tational Linguistics, COLING '10, 2010.","title":"Large scale parallel document mining for machine translation","authors":"Jakob Uszkoreit, Jay M Ponte, C Ashok, Moshe Popat, Dubiner","year":"2010","venue":"Proceedings of the 23rd International Conference on Computational Linguistics, COLING '10","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Utiyama and Isahara, 2003] Masao Utiyama and Hitoshi Isahara. Reliable measures for aligning japanese-english news articles and sentences. In Proceedings of the 41st An- nual Meeting on Association for Computational Linguis- tics, pages 72-79. Association for Computational Linguis- tics, 2003.","title":"Reliable measures for aligning japanese-english news articles and sentences","authors":"Masao Utiyama, Hitoshi Isahara","year":"2003","venue":"Proceedings of the 41st Annual Meeting on Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Vanderkam et al., 2013] Dan Vanderkam, Rob Schon- berger, Henry Rowley, and Sanjiv Kumar. Nearest neighbor search in google correlate. Technical report, Google, 2013.","title":"Nearest neighbor search in google correlate","authors":"Dan Vanderkam, Rob Schonberger, Henry Rowley, Sanjiv Kumar","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Sys- tems, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wang et al., 2018a] Feng Wang, Jian Cheng, Weiyang Liu, and Haijun Liu. Additive margin softmax for face veri- fication. IEEE Signal Processing Letters, 25(7):926-930, 2018.","title":"Additive margin softmax for face verification","authors":"Wang","year":"2018","venue":"IEEE Signal Processing Letters","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wang et al., 2018b] Wei Wang, Taro Watanabe, Macduff Hughes, Tetsuji Nakagawa, and Ciprian Chelba. Denois- ing neural machine translation training with trusted data and online data selection. In Proceedings of the Third Con- ference on Machine Translation, pages 133-143. Associa- tion for Computational Linguistics, 2018.","title":"Denoising neural machine translation training with trusted data and online data selection","authors":"Wang","year":"2018","venue":"Proceedings of the Third Conference on Machine Translation","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Yang and Li, 2002] Christopher C Yang and Kar Wing Li. Mining english/chinese parallel documents from the world wide web. In Proceedings of the 11th International World Wide Web Conference, Honolulu, USA, 2002.","title":"Mining english/chinese parallel documents from the world wide web","authors":"C Christopher, Kar Yang, Li Wing","year":"2002","venue":"Proceedings of the 11th International World Wide Web Conference","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Yang et al., 2018] Yinfei Yang, Steve Yuan, Daniel Cer, Sheng-Yi Kong, Noah Constant, Petr Pilar, Heming Ge, Yun-hsuan Sung, Brian Strope, and Ray Kurzweil. Learn- ing semantic textual similarity from conversations. In Pro- ceedings of The Third Workshop on Representation Learn- ing for NLP. Association for Computational Linguistics, 2018.","title":"Learning semantic textual similarity from conversations","authors":"Yang","year":"2018","venue":"Proceedings of The Third Workshop on Representation Learning for NLP","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Yin et al., 2015] Wenpeng Yin, Hinrich Schütze, Bing Xi- ang, and Bowen Zhou. Abcnn: Attention-based convolu- tional neural network for modeling sentence pairs. arXiv preprint arXiv:1512.05193, 2015.","title":"Abcnn: Attention-based convolutional neural network for modeling sentence pairs","authors":"Yin","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Ziemski et al., 2016] Michal Ziemski, Marcin Junczys- Dowmunt, and Bruno Pouliquen. The united nations par- allel corpus v1. 0. April 2016.","title":"The united nations parallel corpus v1","authors":"Michal Ziemski, Marcin Junczys-Dowmunt, Bruno Pouliquen","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zweigenbaum et al., 2018] Pierre Zweigenbaum, Serge Sharoff, and Reinhard Rapp. Overview of the third bucc shared task: Spotting parallel sentences in comparable corpora. In Reinhard Rapp, Pierre Zweigenbaum, and Serge Sharoff, editors, Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), May 2018.","title":"Overview of the third bucc shared task: Spotting parallel sentences in comparable corpora","authors":"Zweigenbaum","year":"2018","venue":"Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}