{"type":"ScholarlyArticle","title":"From Feature to Paradigm: Deep Learning in Machine Translation (Extended Abstract)","authors":[{"name":"Marta R. Costa-jussà","slug":"marta-r-costa-jussa","given":"Marta R.","family":"Costa-jussà","affiliation":"TALP Research Center - Universitat Politècnica de Catalunya , Barcelona"}],"abstract":"In the last years, deep learning algorithms have highly revolutionized several areas including speech, image and natural language processing. The specific field of Machine Translation (MT) has not remained invariant. Integration of deep learning in MT varies from re-modeling existing features into standard statistical systems to the development of a new architecture. Among the different neural networks, research works use feed-forward neural networks, recurrent neural networks and the encoder-decoder schema. These architectures are able to tackle challenges as having low-resources or morphology variations.\r\n\r\nThis extended abstract focuses on describing the foundational works on the neural MT approach; mentioning its strengths and weaknesses; and including an analysis of the corresponding challenges and future work. The full manuscript [Costa-jussà, 2018] describes, in addition, how these neural networks have been integrated to enhance different aspects and models from statistical MT, including language modeling, word alignment, translation, reordering, and rescoring; and on describing the new neural MT approach together with recent approaches on using subword, characters and training with multilingual\r\nlanguages, among others.","doi":"10.24963/ijcai.2018/789","url":"https://ijcai.org/proceedings/2018/789","pdf":"https://ijcai.org/proceedings/2018/0789.pdf","markdown":"https://ijcai.org/proceedings/2018/789.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/789","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Journal track","track":"Journal track","number":789,"pages":{"first":5583,"last":5587},"keywords":["Computer Vision: Statistical Methods and Machine Learning","Natural Language Processing: Machine Translation","Machine Learning: Deep Learning"],"videos":[],"references":[{"position":1,"raw":"[Aldón, 2016] David Aldón. Sistema de Traducción Neu- ronal Usando Bitmaps. B.s. thesis, Universitat Politècnica de Catalunya, 2016.","title":"Sistema de Traducción Neuronal Usando Bitmaps","authors":"David Aldón","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Artetxe et al., 2017] Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. Unsupervised neural ma- chine translation. CoRR, abs/1710.11041, 2017.","title":"Unsupervised neural machine translation","authors":"Mikel Artetxe, Gorka Labaka, Eneko Agirre, Kyunghyun Cho","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bahdanau et al., 2015] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine transla- tion by jointly learning to align and translate. CoRR, abs/1409.0473, 2015.","title":"Neural machine translation by jointly learning to align and translate","authors":"Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bengio, 2009] Yoshua Bengio. Learning deep architectures for ai. Found. Trends Mach. Learn., 2(1):1-127, January 2009.","title":"Learning deep architectures for ai","authors":"Yoshua Bengio","year":"2009","venue":"Found. Trends Mach. Learn","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bojar et al., 2016] Ondřej Bojar, Rajen Chatterjee, Chris- tian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Lo- gacheva, Christof Monz, Matteo Negri, Aurelie Neveol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin Ver- spoor, and Marcos Zampieri. Findings of the 2016 con- ference on machine translation. In Proceedings of the First Conference on Machine Translation, pages 131-198, Berlin, Germany, August 2016. Association for Computa- tional Linguistics.","title":"Findings of the 2016 conference on machine translation","authors":"Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurelie Neveol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin Verspoor, Marcos Zampieri","year":"2016","venue":"Proceedings of the First Conference on Machine Translation","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Castaño and Casacuberta, 1997] M. A. Castaño and F. Casacuberta. A connectionist approach to mt. In Proc. of the EUROSPEECH Conference, 1997.","title":"A connectionist approach to mt","authors":"M A Castaño, F Casacuberta","year":"1997","venue":"Proc. of the EUROSPEECH Conference","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Cho et al., 2014a] Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. On the proper- ties of neural machine translation: Encoder-decoder ap- proaches. In Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation, pages 103-111, Doha, Qatar, October 2014. Association for Computational Linguistics.","title":"On the properties of neural machine translation: Encoder-decoder approaches","authors":"Cho","year":"2014","venue":"Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Cho et al., 2014b] Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the 2014 Confer- ence on Empirical Methods in Natural Language Process- ing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL, pages 1724-1734, 2014. [Costa-jussà and Fonollosa, 2016]","title":"Learning phrase representations using RNN encoder-decoder for statistical machine translation","authors":"Cho","year":"2014","venue":"Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"Marta R. Costa-jussà and José A. R. Fonollosa. Character-based neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 357-361, Berlin, Germany, August 2016. Association for Computational Linguistics.","title":"Character-based neural machine translation","authors":"Marta R Costa-Jussà, A R José, Fonollosa","year":"2016","venue":"Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Costa-jussà et al., 2017] Marta R. Costa-jussà, David Aldón, and José A. R. Fonollosa. Chinese-spanish neural machine translation enhanced with character andword bitmap fonts. Machine Translation, page Accepted for publication, 2017.","title":"Chinese-spanish neural machine translation enhanced with character andword bitmap fonts","authors":"Marta R Costa-Jussà, David Aldón, José A R Fonollosa","year":"2017","venue":"Machine Translation","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Costa-jussà, 2015] Marta R. Costa-jussà. How much hy- bridization does machine translation need? Journal of the Association for Information Science and Technology, 66(10):2160-2165, 2015.","title":"How much hybridization does machine translation need","authors":"Marta R Costa-Jussà","year":"2015","venue":"Journal of the Association for Information Science and Technology","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Costa-jussá, 2017] Marta R. Costa-jussá. Why catalan- spanish neural machine translation? analysis, comparison and combination with standard rule and phrase-based tech- nologies. In Proceedings of the EACL Workshop: Vardial, Valencia, April 2017.","title":"Why catalanspanish neural machine translation? analysis, comparison and combination with standard rule and phrase-based technologies","authors":"Marta R Costa-Jussá","year":"2017","venue":"Proceedings of the EACL Workshop: Vardial","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Costa-jussà, 2018] M.R. Costa-jussà. From feature to paradigm: Deep learning in machine translation. Journal of Artificial Intelligence Research (JAIR), 61, 2018.","title":"From feature to paradigm: Deep learning in machine translation","authors":"M R Costa-Jussà","year":"2018","venue":"Journal of Artificial Intelligence Research (JAIR)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Crego et al., 2016] Josep Maria Crego, Jungi Kim, Guil- laume Klein, Anabel Rebollo, Kathy Yang, Jean Senel- lart, Egor Akhanov, and Patrice Brunelle et al. Sys- tran's pure neural machine translation systems. CoRR, abs/1610.05540, 2016.","title":"Systran's pure neural machine translation systems","authors":"Maria Josep, Jungi Crego, Guillaume Kim, Anabel Klein, Kathy Rebollo, Jean Yang, Egor Senellart, Patrice Akhanov, Brunelle","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Elliott et al., 2015] Desmond Elliott, Stella Frank, and Eva Hasler. Multi-language image description with neural se- quence models. CoRR, abs/1510.04709, 2015.","title":"Multi-language image description with neural sequence models","authors":"Desmond Elliott, Stella Frank, Eva Hasler","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Elman, 1990] Jeffrey L. Elman. Finding structure in time. Cognitive Science, 14(2):179 -211, 1990.","title":"Finding structure in time","authors":"Jeffrey L Elman","year":"1990","venue":"Cognitive Science","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Firat et al., 2017] Orhan Firat, Kyunghyun Cho, Baskaran Sankaran, Fatos T. Yarman Vural, and Yoshua Bengio. Multi-Way, Multilingual Neural Machine Translation. Ac- cepted for publication in Computer Speech and Language, Special Issue in Deep learning for Machine Translation, 2017.","title":"Multi-Way, Multilingual Neural Machine Translation","authors":"Orhan Firat, Kyunghyun Cho, Baskaran Sankaran, T Yarman Fatos, Yoshua Vural, Bengio","year":"2017","venue":"Accepted for publication in Computer Speech and Language, Special Issue in Deep learning for Machine Translation","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Forcada and Ñeco, 1997] Mikel L. Forcada and Ramón P. Ñeco. Recursive hetero-associative memories for transla- tion. In Proceedings of the International Work-Conference on Artificial and Natural Neural Networks: Biological and Artificial Computation: From Neuroscience to Technol- ogy, IWANN '97, pages 453-462, London, UK, UK, 1997. Springer-Verlag.","title":"Recursive hetero-associative memories for translation","authors":"Mikel L Forcada, Ramón P Ñeco","year":"1997","venue":"Proceedings of the International Work-Conference on Artificial and Natural Neural Networks: Biological and Artificial Computation: From Neuroscience to Technology, IWANN '97","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Goodfellow et al., 2016] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016.","title":"Deep Learning","authors":"Ian Goodfellow, Yoshua Bengio, Aaron Courville","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Graves et al., 2013] Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. Speech recognition with deep recur- rent neural networks. In 2013 IEEE International Confer- ence on Acoustics, Speech and Signal Processing, pages 6645-6649. IEEE, May 2013.","title":"Speech recognition with deep recurrent neural networks","authors":"Alex Graves, Abdel-Rahman Mohamed, Geoffrey Hinton","year":"2013","venue":"IEEE International Conference on Acoustics, Speech and Signal Processing","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Hinton et al., 2006] Geoffrey E. Hinton, Simon Osindero, and Yee-Whye Teh. A fast learning algorithm for deep belief nets. Neural Comput., 18(7):1527-1554, July 2006.","title":"A fast learning algorithm for deep belief nets","authors":"Geoffrey E Hinton, Simon Osindero, Yee-Whye Teh","year":"2006","venue":"Neural Comput","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Jean et al., 2015] Sébastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. On using very large target vocabulary for neural machine translation. In Proceedings of the 53rd Annual Meeting of the Association for Compu- tational Linguistics and the 7th International Joint Con- ference on Natural Language Processing (Volume 1: Long Papers), pages 1-10, Beijing, China, July 2015. Associa- tion for Computational Linguistics.","title":"On using very large target vocabulary for neural machine translation","authors":"Sébastien Jean, Kyunghyun Cho, Roland Memisevic, Yoshua Bengio","year":"2015","venue":"Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Johnson et al., 2016] Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda B. Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's multilingual neural machine translation system: Enabling zero-shot translation. CoRR, abs/1611.04558, 2016.","title":"Google's multilingual neural machine translation system: Enabling zero-shot translation","authors":"Melvin Johnson, Mike Schuster, Quoc V Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda B Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, Jeffrey Dean","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Kalchbrenner and Blunsom, 2013] Nal Kalchbrenner and Phil Blunsom. Recurrent continuous translation models. In Proceedings of the 2013 Conference on Empirical Meth- ods in Natural Language Processing, pages 1700-1709, Seattle, Washington, USA, October 2013. Association for Computational Linguistics.","title":"Recurrent continuous translation models","authors":"Nal Kalchbrenner, Phil Blunsom","year":"2013","venue":"Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Koehn and Knowles, 2017] Philipp Koehn and REbecca Knowles. Six challenges for neural machine translation. In ACL Workshop on Neural Machine translation, 2017.","title":"Six challenges for neural machine translation","authors":"Knowles Koehn, Philipp Koehn, Rebecca Knowles","year":"2017","venue":"ACL Workshop on Neural Machine translation","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Koehn et al., 2003] Philipp Koehn, Franz Josef Och, and Daniel Marcu. Statistical phrase-based translation. In Pro- ceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology -Volume 1, NAACL '03, pages 48-54, Stroudsburg, PA, USA, 2003. Association for Computational Linguistics.","title":"Statistical phrase-based translation","authors":"Philipp Koehn, Franz, Josef Och, Daniel Marcu","year":"2003","venue":"Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In Proceedings of the 25th International Conference on Neural Information Process- ing Systems, NIPS'12, pages 1097-1105, USA, 2012. Cur- ran Associates Inc. [Lample et al., 2017]","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"Proceedings of the 25th International Conference on Neural Information Processing Systems, NIPS'12","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"Guillaume Lample, Ludovic Denoyer, and Marc'Aurelio Ranzato. Unsupervised machine translation using monolingual corpora only. CoRR, abs/1711.00043, 2017.","title":"Unsupervised machine translation using monolingual corpora only","authors":"Guillaume Lample, Ludovic Denoyer, Marc'aurelio Ranzato","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Lee et al., 2009] Honglak Lee, Roger Grosse, Rajesh Ran- ganath, and Andrew Y. Ng. Convolutional deep belief net- works for scalable unsupervised learning of hierarchical representations. In Proceedings of the 26th Annual In- ternational Conference on Machine Learning, ICML '09, pages 609-616, New York, NY, USA, 2009. ACM.","title":"Convolutional deep belief net-works for scalable unsupervised learning of hierarchical representations","authors":"Lee","year":"2009","venue":"Proceedings of the 26th Annual International Conference on Machine Learning, ICML '09","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Lee et al., 2016] Jason Lee, Kyunghyun Cho, and Thomas Hofmann. Fully character-level neural machine translation without explicit segmentation. CoRR, abs/1610.03017, 2016.","title":"Fully character-level neural machine translation without explicit segmentation","authors":"Jason Lee, Kyunghyun Cho, Thomas Hofmann","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Maegaard, 1989] B. Maegaard. Eurotra: the machine trans- lation project of the european communities. Perspectives in artificial intelligence, II, 1989. [","title":"Eurotra: the machine translation project of the european communities","authors":"B Maegaard","year":"1989","venue":"Perspectives in artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"Philipson, 2014 accessed September 2017]","title":null,"authors":"Philipson","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"J. Philipson. Systran: A brief history of machine translation, 2014; accessed September 2017.","title":"Systran: A brief history of machine translation","authors":"J Philipson","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Pouget-Abadie et al., 2014] Jean Pouget-Abadie, Dzmitry Bahdanau, Bart van Merrienboer, Kyunghyun Cho, and Yoshua Bengio. Overcoming the curse of sentence length for neural machine translation using automatic segmen- tation. In Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation, pages 78-85, Doha, Qatar, October 2014. Association for Computational Linguistics.","title":"Overcoming the curse of sentence length for neural machine translation using automatic segmentation","authors":"Pouget-Abadie","year":"2014","venue":"Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Schwenk et al., 2006] Holger Schwenk, Marta R. Costa- Jussà, and José A. R. Fonollosa. Continuous space lan- guage models for the IWSLT 2006 task. In 2006 Interna- tional Workshop on Spoken Language Translation, IWSLT 2006, Keihanna Science City, Kyoto, Japan, November 27- 28, 2006, pages 166-173, 2006.","title":"Continuous space language models for the IWSLT 2006 task","authors":"Holger Schwenk, Marta R Costa-Jussà, José A R Fonollosa","year":"2006","venue":"2006 International Workshop on Spoken Language Translation, IWSLT 2006","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Sennrich et al., 2016] Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715-1725, Berlin, Ger- many, August 2016. Association for Computational Lin- guistics. [Sutskever et al., 2014] Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural networks. In Advances in Neural Information Process- ing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pages 3104-3112, 2014.","title":"Neural machine translation of rare words with subword units","authors":"Sennrich","year":"2014","venue":"Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Weiss et al., 2017] Ron J. Weiss, Jan Chorowski, Navdeep Jaitly, Yonghui Wu, and Zhifeng Chen. Sequence-to- sequence models can directly translate foreign speech. CoRR, abs/1703.08581, 2017.","title":"Sequence-tosequence models can directly translate foreign speech","authors":"Ron J Weiss, Jan Chorowski, Navdeep Jaitly, Yonghui Wu, Zhifeng Chen","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}