{"type":"ScholarlyArticle","title":"Out of Sight But Not Out of Mind: An Answer Set Programming Based Online Abduction Framework for Visual Sensemaking in Autonomous Driving","authors":[{"name":"Jakob Suchan","slug":"jakob-suchan","given":"Jakob","family":"Suchan","affiliation":"University of Bremen , Germany; CoDesign Lab / Cognitive Vision"},{"name":"Mehul Bhatt","slug":"mehul-bhatt","given":"Mehul","family":"Bhatt","affiliation":"Örebro University , Sweden; CoDesign Lab / Cognitive Vision"},{"name":"Srikrishna Varadarajan","slug":"srikrishna-varadarajan","given":"Srikrishna","family":"Varadarajan","affiliation":"CoDesign Lab / Cognitive Vision"}],"abstract":"We  demonstrate the need and potential of systematically integrated vision and semantics solutions for visual sensemaking (in the backdrop of autonomous driving). \r\nA general method for online visual sensemaking using answer set programming is systematically formalised and fully implemented. The method integrates state of the art in visual computing, and is developed as a modular framework usable within hybrid architectures for perception & control. We evaluate and demo with community established benchmarks KITTIMOD and MOT. As use-case, we focus on the significance of human-centred visual sensemaking ---e.g., semantic representation and explainability, question-answering, commonsense interpolation--- in safety-critical autonomous driving situations.","doi":"10.24963/ijcai.2019/260","url":"https://ijcai.org/proceedings/2019/260","pdf":"https://ijcai.org/proceedings/2019/0260.pdf","markdown":"https://ijcai.org/proceedings/2019/260.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/260","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Knowledge Representation and Reasoning","track":"Main track","number":260,"pages":{"first":1879,"last":1885},"keywords":["Computer Vision: Structural and Model-Based Approaches, Knowledge Representation and Reasoning","Computer Vision: Motion and Tracking","Knowledge Representation and Reasoning: Non-monotonic Reasoning","Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning"],"videos":[],"references":[{"position":1,"raw":"trk(trk_3, car). trk_state(trk_3, active). ... ... trk(trk_41, car).trk_state(trk_41, active). ... ... det(det_1, car, 98). ... box2d(trk_3, 660, 460, 134, 102). ... ... box2d(trk_41, 631, 471, 40, 47). ... ... occurs_at(hides_behind(trk_41,trk_3),179)) ...","title":"trk(trk_41, car).trk_state(trk_41, active)","authors":null,"year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"anticipate(unhides_from_behind(Trk1, Trk2), T) :- time(T), curr_time < T, holds_at(hidden_by(Trk1, Trk2), curr_time), topology(proper_part, Trk1, Trk2), movement(moves_out_of, Trk1, Trk2, T).","title":"anticipate(unhides_from_behind(Trk1, Trk2), T) :time(T), curr_time < T, holds_at(hidden_by(Trk1, Trk2), curr_time), topology(proper_part, Trk1, Trk2), movement(moves_out_of","authors":null,"year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"point2d(interpolated_position(Trk, T), PosX, PosY) :- time(T), curr_time < T, T1 = T-curr_time, box2d(Trk1, X, Y,_,_), trk_mov(Trk1, MovX, MovY), PosX = X+MovX * T1, PosY = Y+MovX * T1. anticipate(unhides_from_behind(trk_41, trk_2), 202)","title":"PosX, PosY) :time(T), curr_time < T, T1 = T-curr_time, box2d(Trk1, X, Y","authors":"T Trk","year":null,"venue":"PosX = X+MovX * T1, PosY = Y+MovX * T1. anticipate","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"point2d(interpolated_position(trk_41, 202), 738, 495)","title":"interpolated_position","authors":null,"year":null,"venue":"interpolated_position","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"warning(hidden_entity_in_front(Trk1, T)) :- time(T), T-curr_time < anticipation_threshold, anticipate(unhides_from_behind(Trk1, _), T), position(in_front, interpolated_pos(Trk1, T)). References","title":"warning(hidden_entity_in_front(Trk1, T)) :time(T), T-curr_time < anticipation_threshold, anticipate(unhides_from_behind(Trk1, _), T), position","authors":null,"year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Aditya et al., 2015] Somak Aditya, Yezhou Yang, Chitta Baral, Cornelia Fermuller, and Yiannis Aloimonos. Visual common- sense for scene understanding using perception, semantic parsing and reasoning. In 2015 AAAI Spring Symposium Series, 2015.","title":"Visual commonsense for scene understanding using perception, semantic parsing and reasoning","authors":"Somak Aditya, Yezhou Yang, Chitta Baral, Cornelia Fermuller, Yiannis Aloimonos","year":"2015","venue":"AAAI Spring Symposium Series","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Allen, 1983] James F. Allen. Maintaining knowledge about tem- poral intervals. Commun. ACM, 26(11):832-843, 1983.","title":"Maintaining knowledge about temporal intervals","authors":"James F Allen","year":"1983","venue":"Commun. ACM","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Balbiani et al., 1999] Philippe Balbiani, Jean-Franc ¸ois Condotta, and Luis Fariñas del Cerro. A new tractable subclass of the rect- angle algebra. In Thomas Dean, editor, IJCAI 1999, Sweden, pages 442-447.","title":"A new tractable subclass of the rectangle algebra","authors":"Philippe Balbiani, Jean-Franc ¸ois Condotta, Luis Fariñas Del Cerro","year":null,"venue":"IJCAI 1999","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"Morgan Kaufmann, 1999.","title":null,"authors":"Morgan Kaufmann","year":"1999","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Bernardin and Stiefelhagen, 2008] Keni Bernardin and Rainer Stiefelhagen. Evaluating multiple object tracking performance: The clear mot metrics. EURASIP Journal on Image and Video Processing, 2008(1):246309, May 2008.","title":"Evaluating multiple object tracking performance: The clear mot metrics","authors":"Stiefelhagen Bernardin, Keni Bernardin, Rainer Stiefelhagen","year":"2008","venue":"EURASIP Journal on Image and Video Processing","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Bewley et al., 2016] Alex Bewley, Zongyuan Ge, Lionel Ott, Fabio Ramos, and Ben Upcroft. Simple online and realtime track- ing. In 2016 IEEE International Conference on Image Processing (ICIP), pages 3464-3468, 2016.","title":"Simple online and realtime tracking","authors":"Alex Bewley, Zongyuan Ge, Lionel Ott, Fabio Ramos, Ben Upcroft","year":"2016","venue":"IEEE International Conference on Image Processing (ICIP)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Bhatt and Loke, 2008] Mehul Bhatt and Seng W. Loke. Modelling dynamic spatial systems in the situation calculus. Spatial Cogni- tion & Computation, 8(1-2):86-130, 2008.","title":"Modelling dynamic spatial systems in the situation calculus","authors":"Mehul Bhatt, Seng W Loke","year":"2008","venue":"Spatial Cognition & Computation","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Bhatt et al., 2013] Mehul Bhatt, Carl Schultz, and Christian Freksa. The 'Space' in Spatial Assistance Systems: Conception, Formalisation and Computation. In: Representing space in cog- nition: Interrelations of behavior, language, and formal models. Series: Explorations in Language and Space. 978-0-19-967991- 1, Oxford University Press, 2013.","title":"The 'Space' in Spatial Assistance Systems: Conception, Formalisation and Computation","authors":"Mehul Bhatt, Carl Schultz, Christian Freksa","year":"2013","venue":"Representing space in cognition: Interrelations of behavior, language, and formal models. Series: Explorations in Language and Space","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Blythe et al., 2011] James Blythe, Jerry R. Hobbs, Pedro Domin- gos, Rohit J. Kate, and Raymond J. Mooney. Implementing weighted abduction in markov logic. In Proc. of 9th Intl. Confer- ence on Computational Semantics, IWCS '11, USA, 2011. ACL.","title":"Implementing weighted abduction in markov logic","authors":"James Blythe, Jerry R Hobbs, Pedro Domingos, Rohit J Kate, Raymond J Mooney","year":"2011","venue":"Proc. of 9th Intl. Conference on Computational Semantics, IWCS '11","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[BMVI, 2018] BMVI. Report by the ethics commission on auto- mated and connected driving., BMVI: Federal ministry of trans- port and digital infrastructure, germany, 2018.","title":"BMVI. Report by the ethics commission on automated and connected driving","authors":"Bmvi","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Brewka et al., 2011] Gerhard Brewka, Thomas Eiter, and Miroslaw Truszczyński. Answer set programming at a glance. Commun. ACM, 54(12):92-103, December 2011.","title":"Answer set programming at a glance","authors":"Gerhard Brewka, Thomas Eiter, Miroslaw Truszczyński","year":"2011","venue":"Commun. ACM","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Chen et al., 2018] Liang-Chieh Chen, Yukun Zhu, George Papan- dreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmen- tation. arXiv:1802.02611, 2018.","title":"Encoder-decoder with atrous separable convolution for semantic image segmentation","authors":"Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, Hartwig Adam","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Davis and Marcus, 2015] Ernest Davis and Gary Marcus. Com- monsense reasoning and commonsense knowledge in artificial intelligence. Commun. ACM, 58(9):92-103, 2015.","title":"Commonsense reasoning and commonsense knowledge in artificial intelligence","authors":"Marcus ; Davis, Ernest Davis, Gary Marcus","year":"2015","venue":"Commun. ACM","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Dubba et al., 2015] Krishna Sandeep Reddy Dubba, Anthony G. Cohn, David C. Hogg, Mehul Bhatt, and Frank Dylla. Learning relational event models from video. J. Artif. Intell. Res. (JAIR), 53:41-90, 2015.","title":"Learning relational event models from video","authors":"Krishna Sandeep, Reddy Dubba, Anthony G Cohn, David C Hogg, Mehul Bhatt, Frank Dylla","year":"2015","venue":"J. Artif. Intell. Res. (JAIR)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Gebser et al., 2012] Martin Gebser, Roland Kaminski, Benjamin Kaufmann, and Torsten Schaub. Answer Set Solving in Practice. Morgan & Claypool Publishers, 2012.","title":"Answer Set Solving in Practice","authors":"Martin Gebser, Roland Kaminski, Benjamin Kaufmann, Torsten Schaub","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Gebser et al., 2014] Martin Gebser, Roland Kaminski, Benjamin Kaufmann, and Torsten Schaub. Clingo = ASP + control: Pre- liminary report. CoRR, abs/1405.3694, 2014.","title":"Clingo = ASP + control: Preliminary report","authors":"Martin Gebser, Roland Kaminski, Benjamin Kaufmann, Torsten Schaub","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Geiger et al., 2012] Andreas Geiger, Philip Lenz, and Raquel Ur- tasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In Conference on Computer Vision and Pattern Recognition (CVPR), 2012.","title":"Are we ready for autonomous driving? the kitti vision benchmark suite","authors":"Andreas Geiger, Philip Lenz, Raquel Urtasun","year":"2012","venue":"Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Hazarika and Cohn, 2002] Shyamanta M Hazarika and Anthony G Cohn. Abducing qualitative spatio-temporal histories from par- tial observations. In KR, pages 14-25, 2002.","title":"Abducing qualitative spatio-temporal histories from partial observations","authors":"Cohn ; Hazarika, M Shyamanta, Anthony G Hazarika, Cohn","year":"2002","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Kautz, 1991] Henry A. Kautz. Reasoning about plans. chapter A Formal Theory of Plan Recognition and Its Implementation, pages 69-124.","title":"Reasoning about plans. chapter A Formal Theory of Plan Recognition and Its Implementation","authors":"Henry A Kautz","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Liu et al., 2016] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott E. Reed, Cheng-Yang Fu, and Alexan- der C. Berg. SSD: single shot multibox detector. In ECCV (1), volume 9905 of LNCS, pages 21-37. Springer, 2016.","title":"SSD: single shot multibox detector","authors":"Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott E Reed, Cheng-Yang Fu, Alexander C Berg","year":"2016","venue":"ECCV (1)","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Ma et al., 2014] Jiefei Ma, Rob Miller, Leora Morgenstern, and Theodore Patkos. An epistemic event calculus for asp-based rea- soning about knowledge of the past, present and future. In LPAR: 19th Intl. Conf. on Logic for Programming, Artificial Intelligence and Reasoning, volume 26 of EPiC Series in Computing, pages 75-87. EasyChair, 2014.","title":"An epistemic event calculus for asp-based reasoning about knowledge of the past, present and future","authors":"Ma","year":"2014","venue":"LPAR: 19th Intl. Conf. on Logic for Programming, Artificial Intelligence and Reasoning","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Mani and Pustejovsky, 2012] Inderjeet Mani and James Puste- jovsky. Interpreting Motion -Grounded Representations for Spa- tial Language, volume 5 of Explorations in language and space. Oxford University Press, 2012.","title":"Inderjeet Mani and James Pustejovsky. Interpreting Motion -Grounded Representations for Spatial Language, volume 5 of Explorations in language and space","authors":"Pustejovsky Mani","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Milan et al., 2016] Anton Milan, Laura Leal-Taixé, Ian D. Reid, Stefan Roth, and Konrad Schindler. MOT16: A benchmark for multi-object tracking. CoRR, abs/1603.00831, 2016.","title":"MOT16: A benchmark for multi-object tracking","authors":"Anton Milan, Laura Leal-Taixé, Ian D Reid, Stefan Roth, Konrad Schindler","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Miller et al., 2013] Rob Miller, Leora Morgenstern, and Theodore Patkos. Reasoning about knowledge and action in an epistemic event calculus. In COMMONSENSE 2013, 2013.","title":"Reasoning about knowledge and action in an epistemic event calculus","authors":"Rob Miller, Leora Morgenstern, Theodore Patkos","year":"2013","venue":"COMMONSENSE 2013","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Muller, 1998] Philippe Muller. A qualitative theory of motion based on spatio-temporal primitives. In Anthony G. Cohn et. al., editor, KR 1998, Italy.","title":"A qualitative theory of motion based on spatio-temporal primitives","authors":"Philippe Muller","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"Morgan Kaufmann, 1998.","title":null,"authors":"Morgan Kaufmann","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Pan et al., 2018] Xingang Pan, Jianping Shi, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Spatial as deep: Spatial CNN for traf- fic scene understanding. In Sheila A. McIlraith and Kilian Q. Weinberger, editors, AAAI 2018. AAAI Press, 2018.","title":"Spatial as deep: Spatial CNN for traffic scene understanding","authors":"Xingang Pan, Jianping Shi, Ping Luo, Xiaogang Wang, Xiaoou Tang","year":"2018","venue":"AAAI 2018","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Redmon and Farhadi, 2018] Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement. CoRR, abs/1804.02767, 2018.","title":"Yolov3: An incremental improvement","authors":"Joseph Redmon, Ali Farhadi","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Ren et al., 2015] Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. Faster R-CNN: towards real-time object detection with region proposal networks. In Annual Conference on Neural Information Processing Systems 2015, Canada, 2015.","title":"Faster R-CNN: towards real-time object detection with region proposal networks","authors":"Ren","year":"2015","venue":"Annual Conference on Neural Information Processing Systems 2015","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Shanahan, 2005] Murray Shanahan. Perception as abduction: Turning sensor data into meaningful representation. Cognitive Science, 29(1):103-134, 2005.","title":"Perception as abduction: Turning sensor data into meaningful representation","authors":"Murray Shanahan, Shanahan","year":"2005","venue":"Cognitive Science","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Suchan and Bhatt, 2016] Jakob Suchan and Mehul Bhatt. Seman- tic question-answering with video and eye-tracking data: AI foundations for human visual perception driven cognitive film studies. In S. Kambhampati, editor, IJCAI 2016, New York, USA, pages 2633-2639. IJCAI/AAAI Press, 2016.","title":"Semantic question-answering with video and eye-tracking data: AI foundations for human visual perception driven cognitive film studies","authors":"Bhatt Suchan, Jakob Suchan, Mehul Bhatt","year":"2016","venue":"IJCAI 2016","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Suchan et al., 2018] Jakob Suchan, Mehul Bhatt, Przemyslaw An- drzej Walega, and Carl P. L. Schultz. Visual explanation by high- level abduction: On answer-set programming driven reasoning about moving objects. In: AAAI 2018. AAAI Press, 2018.","title":"Visual explanation by highlevel abduction: On answer-set programming driven reasoning about moving objects","authors":"Jakob Suchan, Mehul Bhatt, Przemyslaw, Andrzej Walega, Carl P L Schultz","year":"2018","venue":"AAAI 2018","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Tu et al., 2014] Kewei Tu, Meng Meng, Mun Wai Lee, Tae Eun Choe, and Song Chun Zhu. Joint video and text parsing for under- standing events and answering queries. IEEE MultiMedia, 2014.","title":"Joint video and text parsing for understanding events and answering queries","authors":"Kewei Tu, Meng Meng, Mun Wai Lee, Tae, Eun Choe, Song Chun, Zhu","year":"2014","venue":"IEEE MultiMedia","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Yu et al., 2015] Haonan Yu, N. Siddharth, Andrei Barbu, and Jef- frey Mark Siskind. A Compositional Framework for Grounding Language Inference, Generation, and Acquisition in Video. J. Artif. Intell. Res. (JAIR), 52:601-713, 2015.","title":"A Compositional Framework for Grounding Language Inference, Generation, and Acquisition in Video","authors":"Haonan Yu, N Siddharth, Andrei Barbu, Jeffrey Mark Siskind","year":"2015","venue":"J. Artif. Intell. Res. (JAIR)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}