{"type":"ScholarlyArticle","title":"Composing Neural Learning and Symbolic Reasoning with an Application to Visual Discrimination","authors":[{"name":"Adithya Murali","slug":"adithya-murali","given":"Adithya","family":"Murali","affiliation":"Department of Computer Science , University of Illinois at Urbana-Champaign"},{"name":"Atharva Sehgal","slug":"atharva-sehgal","given":"Atharva","family":"Sehgal","affiliation":"Department of Computer Science , University of Texas at Austin"},{"name":"Paul Krogmeier","slug":"paul-krogmeier","given":"Paul","family":"Krogmeier","affiliation":"Department of Computer Science , University of Illinois at Urbana-Champaign"},{"name":"P. Madhusudan","slug":"p-madhusudan","given":"P.","family":"Madhusudan","affiliation":"Department of Computer Science , University of Illinois at Urbana-Champaign"}],"abstract":"We consider the problem of combining machine learning models to perform higher-level cognitive tasks with clear specifications. We propose the novel problem of Visual Discrimination Puzzles (VDP) that requires finding interpretable discriminators that classify images according to a logical specification. Humans can solve these puzzles with ease and they give robust, verifiable, and interpretable discriminators as answers. We propose a compositional neurosymbolic framework that combines a neural network to detect objects and relationships with a symbolic learner that finds interpretable discriminators. We create large classes of VDP datasets involving natural and artificial images and show that our neurosymbolic framework performs favorably compared to several purely neural approaches.","doi":"10.24963/ijcai.2022/466","url":"https://ijcai.org/proceedings/2022/466","pdf":"https://ijcai.org/proceedings/2022/0466.pdf","markdown":"https://ijcai.org/proceedings/2022/466.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/466","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":466,"pages":{"first":3358,"last":3365},"keywords":["AI Ethics, Trust, Fairness: Trustworthy AI","Machine Learning: Few-shot learning","Machine Learning: Explainable/Interpretable Machine Learning","Computer Vision: Visual reasoning and symbolic representation","Machine Learning: Neuro-Symbolic Methods"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986945","length_sec":81,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985088","length_sec":358,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Alur et al., 2015] Rajeev Alur, Rastislav Bodík, Eric Dallal, Dana Fisman, Pranav Garg, Garvit Juniwal, Hadas Kress- Gazit, P. Madhusudan, Milo M. K. Martin, Mukund Raghothaman, Shambwaditya Saha, Sanjit A. Seshia, Rishabh Singh, Armando Solar-Lezama, Emina Torlak, and Abhishek Udupa. Syntax-guided synthesis. Depend- able Software Systems Engineering, 2015.","title":"Syntax-guided synthesis","authors":"Rajeev Alur, Rastislav Bodík, Eric Dallal, Dana Fisman, Pranav Garg, Garvit Juniwal, Hadas Kress-Gazit, P Madhusudan, M K Milo, Mukund Martin, Shambwaditya Raghothaman, Sanjit A Saha, Rishabh Seshia, Armando Singh, Emina Solar-Lezama, Abhishek Torlak, Udupa","year":"2015","venue":"Dependable Software Systems Engineering","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Alur et al., 2018] Rajeev Alur, Rishabh Singh, Dana Fis- man, and Armando Solar-Lezama. Search-based program synthesis. Commun. ACM, 61(12):84-93, November 2018.","title":"Search-based program synthesis","authors":"Rajeev Alur, Rishabh Singh, Dana Fisman, Armando Solar-Lezama","year":"2018","venue":"Commun. ACM","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Amizadeh et al., 2020] Saeed Amizadeh, Hamid Palangi, Oleksandr Polozov, Yichen Huang, and Kazuhito Koishida. Neuro-symbolic visual reasoning: Disentangling \"visual\" from \"reasoning\". ICML, 2020.","title":"Neuro-symbolic visual reasoning: Disentangling \"visual\" from \"reasoning","authors":"Saeed Amizadeh, Hamid Palangi, Oleksandr Polozov, Yichen Huang, Kazuhito Koishida","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Andreas et al., 2016] Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. CVPR, 2016.","title":"Neural module networks","authors":"Jacob Andreas, Marcus Rohrbach, Trevor Darrell, Dan Klein","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Andreas et al., 2018] Jacob Andreas, Dan Klein, and Sergey Levine. Learning with latent language. ACL, 2018.","title":"Learning with latent language","authors":"Jacob Andreas, Dan Klein, Sergey Levine","year":"2018","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Antol et al., 2015] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. ICCV, 2015.","title":"Vqa: Visual question answering","authors":"Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, Devi Parikh","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Balog et al., 2017] Matej Balog, Alexander L. Gaunt, Marc Brockschmidt, Sebastian Nowozin, and Daniel Tarlow. Deepcoder: Learning to write programs. ICLR, 2017.","title":"Deepcoder: Learning to write programs","authors":"Matej Balog, Alexander L Gaunt, Marc Brockschmidt, Sebastian Nowozin, Daniel Tarlow","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Bunel et al., 2018] Rudy Bunel, Matthew J. Hausknecht, Jacob Devlin, Rishabh Singh, and Pushmeet Kohli. Lever- aging grammar and reinforcement learning for neural program synthesis. ICLR, 2018.","title":"Leveraging grammar and reinforcement learning for neural program synthesis","authors":"Rudy Bunel, Matthew J Hausknecht, Jacob Devlin, Rishabh Singh, Pushmeet Kohli","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[de Moura and Bjørner, 2008] Leonardo de Moura and Nikolaj Bjørner. Z3: An efficient smt solver. TACAS, 2008.","title":"Leonardo de Moura and Nikolaj Bjørner. Z3: An efficient smt solver","authors":"De Moura, Bjørner","year":"2008","venue":"TACAS","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Devlin et al., 2017] Jacob Devlin, Jonathan Uesato, Surya Bhupatiraju, Rishabh Singh, Abdel-rahman Mohamed, and Pushmeet Kohli. Robustfill: Neural program learning under noisy i/o. ICML, 2017.","title":"Robustfill: Neural program learning under noisy i/o","authors":"Jacob Devlin, Jonathan Uesato, Surya Bhupatiraju, Rishabh Singh, Abdel-Rahman Mohamed, Pushmeet Kohli","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Ellis et al., 2018] Kevin Ellis, Daniel Ritchie, Armando Solar-Lezama, and Joshua B. Tenenbaum. Learning to infer graphics programs from hand-drawn images. NIPS, 2018.","title":"Learning to infer graphics programs from hand-drawn images","authors":"Kevin Ellis, Daniel Ritchie, Armando Solar-Lezama, Joshua B Tenenbaum","year":"2018","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Enderton, 2011] Herbert B. Enderton. A Mathematical Introduction to Logic. Academic Press, 2011.","title":"A Mathematical Introduction to Logic","authors":"B Herbert, Enderton","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Evans and Grefenstette, 2018] Richard Evans and Edward Grefenstette. Learning explanatory rules from noisy data. J. Artif. Int. Res., 61(1):1-64, Jan 2018.","title":"Learning explanatory rules from noisy data","authors":"Grefenstette Evans, Richard Evans, Edward Grefenstette","year":"2018","venue":"J. Artif. Int. Res","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Goodfellow et al., 2016] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. http://www.deeplearningbook.org.","title":"Deep Learning","authors":"Ian Goodfellow, Yoshua Bengio, Aaron Courville","year":"2016","venue":null,"doi":null,"url":"http://www.deeplearningbook.org","internal_url":null},{"position":15,"raw":"[Gulwani et al., 2015] Sumit Gulwani, José Hernández- Orallo, Emanuel Kitzelmann, Stephen H. Muggleton, Ute Schmid, and Benjamin Zorn. Inductive programming meets the real world. Commun. ACM, 58(11):90-99, October 2015.","title":"Inductive programming meets the real world","authors":"Sumit Gulwani, José Hernández-Orallo, Emanuel Kitzelmann, Stephen H Muggleton, Ute Schmid, Benjamin Zorn","year":"2015","venue":"Commun. ACM","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Hudson and Manning, 2019] Drew A. Hudson and Christo- pher D. Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. CVPR, 2019.","title":"Gqa: A new dataset for real-world visual reasoning and compositional question answering","authors":"Drew A Hudson, Christopher D Manning","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Johnson et al., 2017] J. Johnson, B. Hariharan, Laurens van der Maaten, Li Fei-Fei, C. L. Zitnick, and Ross B. Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. CVPR, 2017.","title":"Clevr: A diagnostic dataset for compositional language and elementary visual reasoning","authors":"J Johnson, B Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C L Zitnick, Ross B Girshick","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Koh et al., 2020] Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In ICML, 2020.","title":"Concept bottleneck models","authors":"Pang Wei Koh, Thao Nguyen, Siang Yew, Stephen Tang, Emma Mussmann, Been Pierson, Percy Kim, Liang","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Lake et al., 2015] Brenden M. Lake, Ruslan Salakhutdinov, and Joshua B. Tenenbaum. Human-level concept learn- ing through probabilistic program induction. Science, 350(6266):1332-1338, 2015.","title":"Human-level concept learning through probabilistic program induction","authors":"M Brenden, Ruslan Lake, Joshua B Salakhutdinov, Tenenbaum","year":"2015","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Liu et al., 2019] Yunchao Liu, Jiajun Wu, Zheng Wu, Daniel Ritchie, William T. Freeman, and Joshua B. Tenenbaum. Learning to describe scenes with programs. ICLR, 2019.","title":"Learning to describe scenes with programs","authors":"Yunchao Liu, Jiajun Wu, Zheng Wu, Daniel Ritchie, William T Freeman, Joshua B Tenenbaum","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Mao et al., 2019] Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B. Tenenbaum, and Jiajun Wu. The Neuro- Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision. ICLR, 2019.","title":"The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision","authors":"Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B Tenenbaum, Jiajun Wu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Murali et al., 2018] Vijayaraghavan Murali, Letao Qi, Swarat Chaudhuri, and Chris Jermaine. Neural sketch learning for conditional program generation. ICLR, 2018.","title":"Neural sketch learning for conditional program generation","authors":"Vijayaraghavan Murali, Letao Qi, Swarat Chaudhuri, Chris Jermaine","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Nédellec, 1998] Claire Nédellec. Inductive logic program- ming, from machine learning to software engineering by f. bergadano and d. gunetti, the mit press, usa, 1996. Knowl. Eng. Rev., 13(2):201-208, jul 1998.","title":"Inductive logic programming, from machine learning to software engineering by f. bergadano and d. gunetti, the mit press, usa","authors":"Claire Nédellec","year":"1996","venue":"Knowl. Eng. Rev","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Parisotto et al., 2017] Emilio Parisotto, Abdel-rahman Mohamed, Rishabh Singh, Lihong Li, Dengyong Zhou, and Pushmeet Kohli. Neuro-symbolic program synthesis. ICLR, 2017.","title":"Neuro-symbolic program synthesis","authors":"Emilio Parisotto, Abdel-Rahman Mohamed, Rishabh Singh, Lihong Li, Dengyong Zhou, Pushmeet Kohli","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Raychev et al., 2019] Veselin Raychev, Martin T. Vechev, and Andreas Krause. Predicting program properties from 'big code'. Commun. ACM, 62(3):99-107, 2019.","title":"Predicting program properties from 'big code","authors":"Veselin Raychev, Martin T Vechev, Andreas Krause","year":"2019","venue":"Commun. ACM","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Reynolds et al., 2019] Andrew Reynolds, Haniel Barbosa, Andres Nötzli, Clark Barrett, and Cesare Tinelli. cvc4sy: Smart and fast term enumeration for syntax-guided syn- thesis. CAV, 2019.","title":"cvc4sy: Smart and fast term enumeration for syntax-guided synthesis","authors":"Andrew Reynolds, Haniel Barbosa, Andres Nötzli, Clark Barrett, Cesare Tinelli","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Santoro et al., 2018] Adam Santoro, Felix Hill, David G. T. Barrett, Ari S. Morcos, and Timothy P. Lillicrap. Measur- ing abstract reasoning in neural networks. ICML, 2018.","title":"Measuring abstract reasoning in neural networks","authors":"Adam Santoro, Felix Hill, G T David, Ari S Barrett, Timothy P Morcos, Lillicrap","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Snell et al., 2017] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. NIPS, 2017.","title":"Prototypical networks for few-shot learning","authors":"Jake Snell, Kevin Swersky, Richard Zemel","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Solar-Lezama et al., 2006] Armando Solar-Lezama, Liviu Tancau, Rastislav Bodik, Sanjit Seshia, and Vi- jay Saraswat. Combinatorial sketching for finite programs. SIGOPS Oper. Syst. Rev., 2006.","title":"Combinatorial sketching for finite programs","authors":"Armando Solar-Lezama, Liviu Tancau, Rastislav Bodik, Sanjit Seshia, Vijay Saraswat","year":"2006","venue":"SIGOPS Oper. Syst. Rev","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Sun et al., 2018] Shao-Hua Sun, Hyeonwoo Noh, Sriram Somasundaram, and Joseph Lim. Neural program synthe- sis from diverse demonstration videos. ICML, 2018.","title":"Neural program synthesis from diverse demonstration videos","authors":"Shao-Hua Sun, Hyeonwoo Noh, Sriram Somasundaram, Joseph Lim","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Wang et al., 2014] Jiang Wang, Yang Song, Thomas Le- ung, Chuck Rosenberg, Jingbin Wang, James Philbin, Bo Chen, and Ying Wu. Learning fine-grained image similarity with deep ranking. CVPR, 2014.","title":"Learning fine-grained image similarity with deep ranking","authors":"Jiang Wang, Yang Song, Thomas Leung, Chuck Rosenberg, Jingbin Wang, James Philbin, Bo Chen, Ying Wu","year":"2014","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Wang et al., 2021] Chien-Yao Wang, Alexey Bochkovskiy, and Hong-Yuan Mark Liao. Scaled-yolov4: Scaling cross stage partial network. CVPR, 2021.","title":"Scaled-yolov4: Scaling cross stage partial network","authors":"Chien-Yao Wang, Alexey Bochkovskiy, Hong-Yuan Mark Liao","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Yi et al., 2018] Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Joshua B. Tenenbaum. Neural-symbolic vqa: Disentangling reasoning from vision and language understanding. NeurIPS, 2018.","title":"Neural-symbolic vqa: Disentangling reasoning from vision and language understanding","authors":"Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, Joshua B Tenenbaum","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zellers et al., 2018] Rowan Zellers, Mark Yatskar, Sam Thomson, and Yejin Choi. Neural motifs: Scene graph parsing with global context. CVPR, 2018.","title":"Neural motifs: Scene graph parsing with global context","authors":"Rowan Zellers, Mark Yatskar, Sam Thomson, Yejin Choi","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Zhang et al., 2019] Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chun Zhu. Raven: A dataset for relational and analogical visual reasoning. CVPR, 2019.","title":"Raven: A dataset for relational and analogical visual reasoning","authors":"Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, Song-Chun Zhu","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}