{"type":"ScholarlyArticle","title":"Improving Information Extraction from Images with Learned Semantic Models","authors":[{"name":"Stephan Baier","slug":"stephan-baier","given":"Stephan","family":"Baier","affiliation":"Ludwig Maximilian University Munich; Data Reply GmbH Munich"},{"name":"Yunpu Ma","slug":"yunpu-ma","given":"Yunpu","family":"Ma","affiliation":"Ludwig Maximilian University Munich; Siemens Corporate Technology Munich"},{"name":"Volker Tresp","slug":"volker-tresp","given":"Volker","family":"Tresp","affiliation":"Ludwig Maximilian University Munich; Siemens Corporate Technology Munich"}],"abstract":"Many applications require an understanding of an image that goes beyond the simple detection and classification of its objects. In particular, a great deal  of semantic information is carried in the relationships between objects. We have previously shown, that the combination of a visual model and a statistical semantic prior model can improve on the task of mapping images to their associated scene description. In this paper, we review the model and compare it to a novel conditional multi-way model for visual relationship detection, which does not include an explicitly trained visual prior model. We also discuss potential relationships between the proposed methods and memory models of the human brain.","doi":"10.24963/ijcai.2018/724","url":"https://ijcai.org/proceedings/2018/724","pdf":"https://ijcai.org/proceedings/2018/0724.pdf","markdown":"https://ijcai.org/proceedings/2018/724.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/724","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Sister Conferences Best Papers","track":"Best Sister Conferences","number":724,"pages":{"first":5214,"last":5218},"keywords":["Machine Learning: Machine Learning","Natural Language Processing: Information Extraction"],"videos":[],"references":[{"position":1,"raw":"[Baier et al., 2017] Stephan Baier, Yunpu Ma, and Volker Tresp. Improving visual relationship detection using se- mantic modeling of scene descriptions. International Se- mantic Web Conference, 2017.","title":"Improving visual relationship detection using semantic modeling of scene descriptions","authors":"Stephan Baier, Yunpu Ma, Volker Tresp","year":"2017","venue":"International Semantic Web Conference","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bloehdorn et al., 2005] Stephan Bloehdorn, Kosmas Petridis, Carsten Saathoff, Nikos Simou, Vassilis Tzou- varas, Yannis Avrithis, Siegfried Handschuh, Yiannis Kompatsiaris, Steffen Staab, and Michael G Strintzis. Semantic annotation of images and videos for multimedia analysis. In European Semantic Web Conference, pages 592-607. Springer, 2005.","title":"Semantic annotation of images and videos for multimedia analysis","authors":"Stephan Bloehdorn, Kosmas Petridis, Carsten Saathoff, Nikos Simou, Vassilis Tzouvaras, Yannis Avrithis, Siegfried Handschuh, Yiannis Kompatsiaris, Steffen Staab, Michael G Strintzis","year":"2005","venue":"European Semantic Web Conference","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bordes et al., 2013] Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. In Advances in neural information processing systems, pages 2787-2795, 2013.","title":"Translating embeddings for modeling multi-relational data","authors":"Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, Oksana Yakhnenko","year":"2013","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dong et al., 2014] Xin Dong, Evgeniy Gabrilovich, Geremy Heitz, Wilko Horn, Ni Lao, Kevin Murphy, Thomas Strohmann, Shaohua Sun, and Wei Zhang. Knowledge vault: A web-scale approach to proba- bilistic knowledge fusion. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 601-610. ACM, 2014.","title":"Knowledge vault: A web-scale approach to probabilistic knowledge fusion","authors":"Xin Dong, Evgeniy Gabrilovich, Geremy Heitz, Wilko Horn, Ni Lao, Kevin Murphy, Thomas Strohmann, Shaohua Sun, Wei Zhang","year":"2014","venue":"Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Girshick et al., 2014] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 580-587, 2014.","title":"Rich feature hierarchies for accurate object detection and semantic segmentation","authors":"Ross Girshick, Jeff Donahue, Trevor Darrell, Jitendra Malik","year":"2014","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Griffiths et al., 2008] Thomas L Griffiths, Charles Kemp, and Joshua B Tenenbaum. Bayesian models of cognition. The Cambridge Handbook of Computational Psychology, 2008.","title":"Bayesian models of cognition","authors":"Charles Thomas L Griffiths, Joshua B Kemp, Tenenbaum","year":"2008","venue":"The Cambridge Handbook of Computational Psychology","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Knill and Pouget, 2004] David C Knill and Alexandre Pouget. The bayesian brain: the role of uncertainty in neu- ral coding and computation. TRENDS in Neurosciences, 27(12):712-719, 2004.","title":"The bayesian brain: the role of uncertainty in neural coding and computation","authors":"C David, Alexandre Knill, Pouget","year":"2004","venue":"TRENDS in Neurosciences","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Lu et al., 2016] Cewu Lu, Ranjay Krishna, Michael Bern- stein, and Li Fei-Fei. Visual relationship detection with language priors. In European Conference on Computer Vision, pages 852-869. Springer, 2016.","title":"Visual relationship detection with language priors","authors":"Lu","year":"2016","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[McClelland et al., 1995] James L McClelland, Bruce L Mc- Naughton, and Randall C O'reilly. Why there are comple- mentary learning systems in the hippocampus and neocor- tex: insights from the successes and failures of connection- ist models of learning and memory. Psychological review, 102(3):419, 1995.","title":"Why there are complementary learning systems in the hippocampus and neocor-tex: insights from the successes and failures of connectionist models of learning and memory","authors":"Bruce L James L Mcclelland, Randall C O' Mc-Naughton, Reilly","year":"1995","venue":"Psychological review","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Nickel et al., 2011] Maximilian Nickel, Volker Tresp, and Hans-Peter Kriegel. A three-way model for collective learning on multi-relational data. In Proceedings of the 28th international conference on machine learning (ICML-11), pages 809-816, 2011.","title":"A three-way model for collective learning on multi-relational data","authors":"Maximilian Nickel, Hans-Peter Volker Tresp, Kriegel","year":"2011","venue":"Proceedings of the 28th international conference on machine learning (ICML-11)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Nickel et al., 2016] Maximilian Nickel, Kevin Murphy, Volker Tresp, and Evgeniy Gabrilovich. A review of re- lational machine learning for knowledge graphs. Proceed- ings of the IEEE, 104(1):11-33, 2016.","title":"A review of relational machine learning for knowledge graphs","authors":"Maximilian Nickel, Kevin Murphy, Evgeniy Volker Tresp, Gabrilovich","year":"2016","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Sadeghi and Farhadi, 2011] Mohammad Amin Sadeghi and Ali Farhadi. Recognition using visual phrases. In Com- puter Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, pages 1745-1752. IEEE, 2011.","title":"Recognition using visual phrases","authors":"Mohammad Amin, Sadeghi, Ali Farhadi","year":"2011","venue":"Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Serafini et al., 2017] Luciano Serafini, Ivan Donadello, and Artur d'Avila Garcez. Learning and reasoning in logic ten- sor networks: theory and application to semantic image in- terpretation. In Proceedings of the Symposium on Applied Computing, pages 125-130. ACM, 2017.","title":"Learning and reasoning in logic tensor networks: theory and application to semantic image interpretation","authors":"Serafini","year":"2017","venue":"Proceedings of the Symposium on Applied Computing","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Tresp and Ma, 2016] Volker Tresp and Yunpu Ma. The ten- sor memory hypothesis. NIPS 2016 Workshop on Rep- resentation Learning in Artificial and Biological Neural Networks (MLINI 2016), 2016.","title":"Yunpu Ma. The tensor memory hypothesis","authors":"Ma ; Tresp, Volker Tresp","year":"2016","venue":"NIPS 2016 Workshop on Representation Learning in Artificial and Biological Neural Networks","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Tresp et al., 2015] Volker Tresp, Cristóbal Esteban, Yin- chong Yang, Stephan Baier, and Denis Krompaß. Learning with memory embeddings. arXiv preprint arXiv:1511.07972, 2015.","title":"Learning with memory embeddings","authors":"Cristóbal Volker Tresp, Yinchong Esteban, Stephan Yang, Denis Baier, Krompaß","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Tresp et al., 2017] Volker Tresp, Yunpu Ma, Stephan Baier, and Yinchong Yang. Embedding learning for declarative memories. In European Semantic Web Conference, pages 202-216. Springer, 2017.","title":"Embedding learning for declarative memories","authors":"Yunpu Volker Tresp, Stephan Ma, Yinchong Baier, Yang","year":"2017","venue":"European Semantic Web Conference","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Trouillon et al., 2016] Théo Trouillon, Johannes Welbl, Se- bastian Riedel, Éric Gaussier, and Guillaume Bouchard. Complex embeddings for simple link prediction. In Inter- national Conference on Machine Learning, pages 2071- 2080, 2016.","title":"Complex embeddings for simple link prediction","authors":"Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, Guillaume Bouchard","year":"2016","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Tulving, 2002] Endel Tulving. Episodic memory: From mind to brain. Annual review of psychology, 53(1):1-25, 2002.","title":"Episodic memory: From mind to brain","authors":"Endel Tulving, Tulving","year":"2002","venue":"Annual review of psychology","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Yang et al., 2014] Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. Embedding entities and relations for learning and inference in knowledge bases. arXiv preprint arXiv:1412.6575, 2014.","title":"Embedding entities and relations for learning and inference in knowledge bases","authors":"Bishan Yang, Wen-Tau Yih, Xiaodong He, Jianfeng Gao, Li Deng","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Zhang et al., 2017] Hanwang Zhang, Zawlin Kyaw, Shih- Fu Chang, and Tat-Seng Chua. Visual translation embed- ding network for visual relation detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, volume 2, page 4, 2017.","title":"Visual translation embedding network for visual relation detection","authors":"Hanwang Zhang, Zawlin Kyaw, Shih-Fu Chang, Tat-Seng Chua","year":"2017","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}