{"type":"ScholarlyArticle","title":"From Words to Sentences: A Progressive Learning Approach for Zero-resource Machine Translation with Visual Pivots","authors":[{"name":"Shizhe Chen","slug":"shizhe-chen","given":"Shizhe","family":"Chen","affiliation":"Renmin University of China , Beijing , P.R. China"},{"name":"Qin Jin","slug":"qin-jin","given":"Qin","family":"Jin","affiliation":"Renmin University of China , Beijing , P.R. China"},{"name":"Jianlong Fu","slug":"jianlong-fu","given":"Jianlong","family":"Fu","affiliation":"Microsoft Research Asia , Beijing , P.R. China"}],"abstract":"The neural machine translation model has suffered from the lack of large-scale parallel corpora. In contrast, we humans can learn multi-lingual translations even without parallel texts by referring our languages to the external world. To mimic such human learning behavior, we employ images as pivots to enable zero-resource translation learning. However, a picture tells a thousand words, which makes multi-lingual sentences pivoted by the same image noisy as mutual translations and thus hinders the translation model learning. In this work, we propose a progressive learning approach for image-pivoted zero-resource machine translation. Since words are less diverse when grounded in the image, we first learn word-level translation with image pivots, and then progress to learn the sentence-level translation by utilizing the learned word translation to suppress noises in image-pivoted multi-lingual sentences. Experimental results on two widely used image-pivot translation datasets, IAPR-TC12 and Multi30k, show that the proposed approach significantly outperforms other state-of-the-art methods.","doi":"10.24963/ijcai.2019/685","url":"https://ijcai.org/proceedings/2019/685","pdf":"https://ijcai.org/proceedings/2019/0685.pdf","markdown":"https://ijcai.org/proceedings/2019/685.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/685","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Natural Language Processing","track":"Main track","number":685,"pages":{"first":4932,"last":4938},"keywords":["Computer Vision: Language and Vision","Natural Language Processing: Machine Translation"],"videos":[],"references":[{"position":1,"raw":"[Artetxe et al., 2018] Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. Unsupervised neural ma- chine translation. ICLR, 2018.","title":"Unsupervised neural machine translation","authors":"Mikel Artetxe, Gorka Labaka, Eneko Agirre, Kyunghyun Cho","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bahdanau et al., 2015] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. ICLR, 2015.","title":"Neural machine translation by jointly learning to align and translate","authors":"Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Calixto et al., 2017] Iacer Calixto, Daniel Stein, Evgeny Matusov, Pintu Lohar, Sheila Castilho, and Andy Way. Using images to improve machine-translating e-commerce product listings. In EACL, pages 637-643, 2017.","title":"Using images to improve machine-translating e-commerce product listings","authors":"Calixto","year":"2017","venue":"EACL","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Castilho et al., 2017] Sheila Castilho, Joss Moorkens, Fed- erico Gaspari, Iacer Calixto, John Tinsley, and Andy Way. Is neural machine translation the new state of the art? The Prague Bulletin of Mathematical Linguistics, 108(1):109- 120, 2017.","title":"Is neural machine translation the new state of the art?","authors":"Sheila Castilho, Joss Moorkens, Federico Gaspari, Iacer Calixto, John Tinsley, Andy Way","year":"2017","venue":"The Prague Bulletin of Mathematical Linguistics","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2017] Yun Chen, Yang Liu, Yong Cheng, and Victor OK Li. A teacher-student framework for zero- resource neural machine translation. In ACL, pages 1925- 1935, 2017.","title":"A teacher-student framework for zeroresource neural machine translation","authors":"Yun Chen, Yang Liu, Yong Cheng, O K Victor, Li","year":"2017","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen et al., 2018] Yun Chen, Yang Liu, and Victor OK Li. Zero-resource neural machine translation with multi-agent communication game. AAAI, 2018.","title":"Zero-resource neural machine translation with multi-agent communication game","authors":"Yun Chen, Yang Liu, O K Victor, Li","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chen et al., 2019] Shizhe Chen, Jin Qin, and Alexander Hauptmann. Unsupervised bilingual lexicon induction with mono-lingual multimodal data. AAAI, 2019.","title":"Unsupervised bilingual lexicon induction with mono-lingual multimodal data","authors":"Chen","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Cho et al., 2014] Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the proper- ties of neural machine translation: Encoder-decoder ap- proaches. arXiv preprint arXiv:1409.1259, 2014.","title":"On the properties of neural machine translation: Encoder-decoder approaches","authors":"Cho","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Elliott et al., 2016] Desmond Elliott, Stella Frank, Khalil Sima'an, and Lucia Specia. Multi30k: Multilin- gual english-german image descriptions. arXiv preprint arXiv:1605.00459, 2016.","title":"Multi30k: Multilingual english-german image descriptions","authors":"Desmond Elliott, Stella Frank, Khalil Sima'an, Lucia Specia","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Grubinger et al., 2006] Michael Grubinger, Paul Clough, Henning Müller, and Thomas Deselaers. The iapr tc-12 benchmark: A new evaluation resource for visual infor- mation systems. In Int. Workshop OntoImage, volume 5, 2006.","title":"The iapr tc-12 benchmark: A new evaluation resource for visual information systems","authors":"Michael Grubinger, Paul Clough, Henning Müller, Thomas Deselaers","year":"2006","venue":"Int. Workshop OntoImage","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-1780, 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Johnson et al., 2017] Melvin Johnson, Mike Schuster, Quoc V Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, et al. Google's multilingual neural machine translation system: Enabling zero-shot translation. TACL, 5(1):339-351, 2017.","title":"Google's multilingual neural machine translation system: Enabling zero-shot translation","authors":"Melvin Johnson, Mike Schuster, V Quoc, Maxim Le, Yonghui Krikun, Zhifeng Wu, Nikhil Chen, Fernanda Thorat, Martin Viégas, Greg Wattenberg, Corrado","year":"2017","venue":"TACL","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kiela et al., 2015] Douwe Kiela, Ivan Vulić, and Stephen Clark. Visual bilingual lexicon induction with transferred convnet features. In EMNLP, pages 148-158, 2015.","title":"Visual bilingual lexicon induction with transferred convnet features","authors":"Kiela","year":"2015","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Koehn et al., 2007] Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, et al. Moses: Open source toolkit for statis- tical machine translation. In ACL, pages 177-180, 2007.","title":"Moses: Open source toolkit for statistical machine translation","authors":"Koehn","year":"2007","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kusner et al., 2015] Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Weinberger. From word embeddings to docu- ment distances. In ICML, pages 957-966, 2015.","title":"From word embeddings to document distances","authors":"Matt Kusner, Yu Sun, Nicholas Kolkin, Kilian Weinberger","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lample et al., 2018a] Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc'Aurelio Ranzato. Unsuper- vised machine translation using monolingual corpora only. ICLR, 2018.","title":"Unsupervised machine translation using monolingual corpora only","authors":"Lample","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lample et al., 2018b] Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, et al. Phrase-based & neu- ral unsupervised machine translation. In EMNLP, pages 5039-5049, 2018.","title":"Phrase-based & neural unsupervised machine translation","authors":"Lample","year":"2018","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lee et al., 2018] Jason Lee, Kyunghyun Cho, Jason We- ston, and Douwe Kiela. Emergent translation in multi- agent communication. ICLR, 2018.","title":"Emergent translation in multiagent communication","authors":"Jason Lee, Kyunghyun Cho, Jason Weston, Douwe Kiela","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Nakayama and Nishida, 2017] Hideki Nakayama and Noriki Nishida. Zero-resource machine translation by multimodal encoder-decoder network with multimedia pivot. Machine Translation, 31(1-2):49-64, 2017.","title":"Zero-resource machine translation by multimodal encoder-decoder network with multimedia pivot","authors":"Nishida Nakayama, Hideki Nakayama, Noriki Nishida","year":"2017","venue":"Machine Translation","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Papineni et al., 2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for au- tomatic evaluation of machine translation. In ACL, pages 311-318, 2002.","title":"Bleu: a method for automatic evaluation of machine translation","authors":"Kishore Papineni, Salim Roukos, Todd Ward, Wei-Jing Zhu","year":"2002","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Pele and Werman, 2009] Ofir Pele and Michael Werman. Fast and robust earth mover's distances. In ICCV, pages 460-467. IEEE, September 2009. Rubner et al., 2000]","title":"Fast and robust earth mover's distances","authors":"Werman Pele, Ofir Pele, Michael Werman","year":"2000","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"Yossi Rubner, Carlo Tomasi, and Leonidas J Guibas. The earth mover's distance as a metric for image retrieval. ICCV, 40(2):99-121, 2000.","title":"The earth mover's distance as a metric for image retrieval","authors":"Yossi Rubner, Carlo Tomasi, Leonidas J Guibas","year":"2000","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Sennrich et al., 2016] Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In ACL, volume 1, pages 1715-1725, 2016.","title":"Neural machine translation of rare words with subword units","authors":"Sennrich","year":"2016","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Sharma et al., 2018] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In ACL, volume 1, pages 2556-2565, 2018.","title":"Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning","authors":"Piyush Sharma, Nan Ding, Sebastian Goodman, Radu Soricut","year":"2018","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Su et al., 2018] Yuanhang Su, Kai Fan, Nguyen Bach, C- C Jay Kuo, and Fei Huang. Unsupervised multi- modal neural machine translation. arXiv preprint arXiv:1811.11365, 2018.","title":"Unsupervised multimodal neural machine translation","authors":"Su","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhou et al., 2018] Mingyang Zhou, Runxiang Cheng, Yong Jae Lee, and Zhou Yu. A visual attention grounding neural model for multimodal machine translation. In EMNLP, pages 3643-3653, 2018.","title":"A visual attention grounding neural model for multimodal machine translation","authors":"Mingyang Zhou, Runxiang Cheng, Yong, Jae Lee, Zhou Yu","year":"2018","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zoph et al., 2016] Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight. Transfer learning for low-resource neu- ral machine translation. In EMNLP, pages 1568-1575, 2016.","title":"Transfer learning for low-resource neural machine translation","authors":"Barret Zoph, Deniz Yuret, Jonathan May, Kevin Knight","year":"2016","venue":"EMNLP","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}