{"type":"ScholarlyArticle","title":"Multimodal News Article Analysis","authors":[{"name":"Arnau Ramisa","slug":"arnau-ramisa","given":"Arnau","family":"Ramisa","affiliation":"Institut de Robòtica i Informàtica Industrial , IRI (UPC-CSIC) , Barcelona , Catalonia , Spain; Wide Eyes Technologies , Barcelona , Catalonia , Spain"}],"abstract":"The intersection of Computer Vision and Natural Language Processing has been a hot topic of research in recent years, with results that were unthinkable only a few years ago. In view of this progress, we want to highlight online news articles as a potential next step for this area of research. The rich interrelations of text, tags, images or videos, as well as a vast corpus of general knowledge are an exciting benchmark for high-capacity models such as the deep neural networks. In this paper we present a series of tasks and baseline approaches to leverage corpus such as the BreakingNews dataset.","doi":"10.24963/ijcai.2017/737","url":"https://ijcai.org/proceedings/2017/737","pdf":"https://ijcai.org/proceedings/2017/0737.pdf","markdown":"https://ijcai.org/proceedings/2017/737.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/737","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Early Career","track":"Early Career","number":737,"pages":{"first":5136,"last":5140},"keywords":["Natural Language Processing: Resources and Evaluation","Machine Learning: Deep Learning","Robotics and Vision: Vision and Perception","Natural Language Processing: Information Retrieval"],"videos":[],"references":[{"position":1,"raw":"[Barnard et al., 2003] Kobus Barnard, Pinar Duygulu, David Forsyth, Nando De Freitas, David Blei, and Michael Jordan. Matching words and pictures. The Journal of Machine Learning Research, 3:1107-1135, 2003.","title":"Matching words and pictures","authors":"Kobus Barnard, Pinar Duygulu, David Forsyth, Nando De Freitas, David Blei, Michael Jordan","year":"2003","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chang et al., 2014] Angel Chang, Manolis Savva, and Christopher Manning. Interactive learning of spatial knowledge for text to 3d scene generation. Sponsor: Idi- bon, page 14, 2014.","title":"Interactive learning of spatial knowledge for text to 3d scene generation","authors":"Chang","year":"2014","venue":"Sponsor: Idibon","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen and Zitnick, 2015] Xinlei Chen and C. Lawrence Zit- nick. Mind's eye: A recurrent visual representation for image caption generation. In CVPR, 2015.","title":"Mind's eye: A recurrent visual representation for image caption generation","authors":"Chen, Zitnick ; Xinlei Chen, C Lawrence Zitnick","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Donahue et al., 2015] Jeff Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venu- gopalan, Kate Saenko, and Trevor Darrell. Long-term re- current convolutional networks for visual recognition and description. In CVPR, 2015.","title":"Long-term recurrent convolutional networks for visual recognition and description","authors":"Jeff Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, Trevor Darrell","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Douze et al., 2011] Matthijs Douze, Arnau Ramisa, and Cordelia Schmid. Combining attributes and fisher vectors for efficient image retrieval. In CVPR, pages 745-752, 2011.","title":"Combining attributes and fisher vectors for efficient image retrieval","authors":"Douze","year":"2011","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Feng and Lapata, 2008] Yansong Feng and Mirella Lapata. Automatic image annotation using auxiliary text informa- tion. In Proceedings of ACL-08: HLT, pages 272-280, Columbus, Ohio, 2008.","title":"Automatic image annotation using auxiliary text information","authors":"Lapata Feng, Yansong Feng, Mirella Lapata","year":"2008","venue":"Proceedings of ACL-08: HLT","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Feng and Lapata, 2010] Yansong Feng and Mirella Lapata. Topic Models for Image Annotation and Text Illustration. Conference of the North American Chapter of the ACL: Human Language Technologies, (June):831-839, 2010.","title":"Topic Models for Image Annotation and Text Illustration","authors":"Lapata Feng, Yansong Feng, Mirella Lapata","year":"2010","venue":"Conference of the North American Chapter of the ACL: Human Language Technologies","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Feng and Lapata, 2013] Yansong Feng and Mirella Lapata. Automatic caption generation for news images. IEEE Transactions on Pattern Analysis and Machine Intelli- gence, 35(4):797-812, 2013.","title":"Automatic caption generation for news images","authors":"Lapata Feng, Yansong Feng, Mirella Lapata","year":"2013","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kalchbrenner et al., 2014] Nal Kalchbrenner, Ed- ward Grefenstette, and Phil Blunsom. A convolutional neural network for modelling sentences. In ACL, 2014.","title":"A convolutional neural network for modelling sentences","authors":"Nal Kalchbrenner, Edward Grefenstette, Phil Blunsom","year":"2014","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kalogerakis et al., 2009] Evangelos Kalogerakis, , Olga Vesselova, James Hays, Alexia Efros, and Aaron Hertzmann. Image sequence geolocation with human travel priors. In ICCV, pages 253-260, 2009.","title":"Image sequence geolocation with human travel priors","authors":"Evangelos Kalogerakis, ,, Olga Vesselova, James Hays, Alexia Efros, Aaron Hertzmann","year":"2009","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Karpathy and Fei-Fei, 2015] Andrej Karpathy and Li Fei- Fei. Deep visual-semantic alignments for generating im- age descriptions. In CVPR, 2015.","title":"Deep visual-semantic alignments for generating image descriptions","authors":"Fei-Fei ; Karpathy, Andrej Karpathy, Li Fei-Fei","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kim, 2014] Yoon Kim. Convolutional neural networks for sentence classification. In EMNLP, 2014.","title":"Convolutional neural networks for sentence classification","authors":"Yoon Kim","year":"2014","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kiros et al., 2015] Ryan Kiros, Ruslan Salakhutdinov, and Richard Zemel. Unifying visual-semantic embeddings with multimodal neural language models. TACL, 2015.","title":"Unifying visual-semantic embeddings with multimodal neural language models","authors":"Ryan Kiros, Ruslan Salakhutdinov, Richard Zemel","year":"2015","venue":"TACL","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kovashka et al., 2015] Adriana Kovashka, Devi Parikh, and Kristen Grauman. Whittlesearch: Interactive image search with relative attribute feedback. International Journal of Computer Vision, pages 1-26, 2015.","title":"Whittlesearch: Interactive image search with relative attribute feedback","authors":"Adriana Kovashka, Devi Parikh, Kristen Grauman","year":"2015","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. ImageNet classification with deep convolutional neural networks. In NIPS, 2012.","title":"ImageNet classification with deep convolutional neural networks","authors":"Krizhevsky","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Le and Mikolov, 2014] Quoc Le and Tomas Mikolov. Dis- tributed representations of sentences and documents. In ICML, 2014.","title":"Quoc Le and Tomas Mikolov. Distributed representations of sentences and documents","authors":"Mikolov Le","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Mikolov et al., 2013] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Dis- tributed representations of words and phrases and their compositionality. In NIPS, 2013.","title":"Distributed representations of words and phrases and their compositionality","authors":"Mikolov","year":"2013","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Ramisa et al., 2016] Arnau Ramisa, Fei Yan, Francesc Moreno-Noguer, and Krystian Mikolajczyk. Breakingnews: Article annotation by image and text processing. arXiv:1603.07141 [cs.CV], 2016.","title":"Breakingnews: Article annotation by image and text processing","authors":"Arnau Ramisa, Fei Yan, Francesc Moreno-Noguer, Krystian Mikolajczyk","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Rasiwasia et al., 2007] Nikhil Rasiwasia, Pedro Moreno, and Nuno Vasconcelos. Bridging the gap: Query by semantic example. IEEE Transactions on Multimedia, 9(5):923-938, 2007.","title":"Bridging the gap: Query by semantic example","authors":"Nikhil Rasiwasia, Pedro Moreno, Nuno Vasconcelos","year":"2007","venue":"IEEE Transactions on Multimedia","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Serdyukov et al., 2009] Pavel Serdyukov, Vanessa Mur- dock, and Roelof Van Zwol. Placing flickr photos on a map. In ACM Conference on Research and Development in Information Retrieval, pages 484-491. ACM, 2009.","title":"Placing flickr photos on a map","authors":"Serdyukov","year":"2009","venue":"ACM Conference on Research and Development in Information Retrieval","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Vinyals et al., 2015] Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neu- ral image caption generator. In CVPR, pages 3156-3164, 2015.","title":"Show and tell: A neural image caption generator","authors":"Oriol Vinyals, Alexander Toshev, Samy Bengio, Dumitru Erhan","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Weyand et al., 2016] Tobias Weyand, Ilya Kostrikov, and James Philbin. Planet -photo geolocation with convolu- tional neural networks. arXiv:1602.05314 [cs.CV], 2016.","title":"Planet -photo geolocation with convolutional neural networks","authors":"Weyand","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yosinski et al., 2014] Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In NIPS, pages 3320-3328, 2014.","title":"How transferable are features in deep neural networks?","authors":"Jason Yosinski, Jeff Clune, Yoshua Bengio, Hod Lipson","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhang et al., 2014] Zhanpeng Zhang, Ping Luo, Chen Loy, and Xiaoou Tang. Facial landmark detection by deep mul- titask learning. In ECCV, 2014.","title":"Facial landmark detection by deep multitask learning","authors":"Zhang","year":"2014","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhou and Luo, 2012] Youjie Zhou and Jiebo Luo. Geo- location inference on news articles via multimodal pLSA. ACM International Conference on Multimedia, page 741, 2012.","title":"Geolocation inference on news articles via multimodal pLSA","authors":"Luo ; ] Youjie Zhou, Jiebo Zhou, Luo","year":"2012","venue":"ACM International Conference on Multimedia","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}