picoTrans: Using Pictures as Input for Machine Translation on Mobile Devices

Finch, Andrew M.; Song, Wei; Tanaka-Ishii, Kumiko; Sumita, Eiichiro

doi:10.5591/978-1-57735-516-8/IJCAI11-435

picoTrans: Using Pictures as Input for Machine Translation on Mobile Devices

Andrew M. Finch, Wei Song, Kumiko Tanaka-Ishii, Eiichiro Sumita

Download BibTeX Cite

Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence. Pages 2614-2618.

https://doi.org/10.5591/978-1-57735-516-8/IJCAI11-435

Abstract

In this paper we present a novel user interface that integrates two popular approaches to language translation for travelers allowing multimodal communication between the parties involved: the picture-book, in which the user simply points to multiple picture icons representing what they want to say, and the statistical machine translation system that can translate arbitrary word sequences. Our prototype system tightly couples both processes within a translation framework that inherits many of the the positive features of both approaches, while at the same time mitigating their main weaknesses. Our system differs from traditional approaches in that its mode of input is a sequence of pictures, rather than text or speech. Text in the source language is generated automatically, and is used as a detailed representation of the intended meaning. The picture sequence which not only provides a rapid method to communicate basic concepts but also gives a `second opinion' on the machine transition output that catches machine translation errors and allows the users to retry the translation, avoiding misunderstandings.

Andrew M. Finch, Wei Song, Kumiko Tanaka-Ishii, and Eiichiro Sumita. 2011. picoTrans: Using Pictures as Input for Machine Translation on Mobile Devices. In Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence, IJCAI-11. IJCAI, 2614–2618. https://doi.org/10.5591/978-1-57735-516-8/IJCAI11-435

@inproceedings{ijcai2011p435,
  title     = {picoTrans: Using Pictures as Input for Machine Translation on Mobile Devices},
  author    = {Finch, Andrew M. and Song, Wei and Tanaka-Ishii, Kumiko and Sumita, Eiichiro},
  booktitle = {Proceedings of the Twenty-Second International Joint Conference on
               Artificial Intelligence, {IJCAI-11}},
  publisher = {AAAI Press/International Joint Conferences on Artificial Intelligence},
  editor    = {Toby Walsh},
  pages     = {2614--2618},
  year      = {2011},
  month     = {7},
  doi       = {10.5591/978-1-57735-516-8/IJCAI11-435},
  url       = {https://doi.org/10.5591/978-1-57735-516-8/IJCAI11-435},
}

TY  - CPAPER
TI  - picoTrans: Using Pictures as Input for Machine Translation on Mobile Devices
AU  - Finch, Andrew M.
AU  - Song, Wei
AU  - Tanaka-Ishii, Kumiko
AU  - Sumita, Eiichiro
T2  - Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence, IJCAI-11
PB  - AAAI Press/International Joint Conferences on Artificial Intelligence
PY  - 2011
SP  - 2614
EP  - 2618
DO  - 10.5591/978-1-57735-516-8/IJCAI11-435
UR  - https://doi.org/10.5591/978-1-57735-516-8/IJCAI11-435
ER  -