{"type":"ScholarlyArticle","title":"Sketch Recognition via Part-based Hierarchical Analogical Learning","authors":[{"name":"Kezhen Chen","slug":"kezhen-chen","given":"Kezhen","family":"Chen","affiliation":"Northwestern University"},{"name":"Ken Forbus","slug":"ken-forbus","given":"Ken","family":"Forbus","affiliation":"Northwestern University"},{"name":"Balaji Vasan Srinivasan","slug":"balaji-vasan-srinivasan","given":"Balaji Vasan","family":"Srinivasan","affiliation":"Adobe Research"},{"name":"Niyati Chhaya","slug":"niyati-chhaya","given":"Niyati","family":"Chhaya","affiliation":"Adobe Research"},{"name":"Madeline Usher","slug":"madeline-usher","given":"Madeline","family":"Usher","affiliation":"Northwestern University"}],"abstract":"Sketch recognition has been studied for decades, but it is far from solved. Drawing styles are highly variable across people and adapting to idiosyncratic visual expressions requires data-efficient learning. Explainability also matters, so that users can see why a system got confused about something. This paper introduces a novel part-based approach for sketch recognition, based on hierarchical analogical learning, a new method to apply analogical learning to qualitative representations. Given a sketched object, our system automatically segments it into parts and constructs multi-level qualitative representations of them. Our approach performs analogical generalization at multiple levels of part descriptions and uses coarse-grained results to guide interpretation at finer levels. Experiments on the Berlin TU dataset and the Coloring Book Objects dataset show that the system can learn explainable models in a data-efficient manner.","doi":"10.24963/ijcai.2023/331","url":"https://ijcai.org/proceedings/2023/331","pdf":"https://ijcai.org/proceedings/2023/0331.pdf","markdown":"https://ijcai.org/proceedings/2023/331.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/331","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Humans and AI","track":"Main Track","number":331,"pages":{"first":2967,"last":2974},"keywords":["Humans and AI: HAI: Cognitive modeling","Knowledge Representation and Reasoning: KRR: Qualitative, geometric, spatial, and temporal reasoning","Knowledge Representation and Reasoning: KRR: Case-based reasoning"],"videos":[],"references":[{"position":1,"raw":"[Biederman, 1987] Irving Biederman. Recognition-by- components: a theory of human image understanding. Psychological Review, 23(2), 1987.","title":"Recognition-bycomponents: a theory of human image understanding","authors":"Irving Biederman, Biederman","year":"1987","venue":"Psychological Review","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Blum, 1967] Harry Blum. A transformation for extracting new descriptors of shape. MIT Press, 1967.","title":"A transformation for extracting new descriptors of shape","authors":"Harry Blum","year":"1967","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen and Forbus, 2018] Kezhen Chen and Kenneth Forbus. Action recognition from skeleton data via analogical gen- eralization over qualitative representations. Proceedings of the AAAI Conference, 32, 2018.","title":"Action recognition from skeleton data via analogical generalization over qualitative representations","authors":"Chen, Forbus ; ] Kezhen Chen, Kenneth Forbus","year":"2018","venue":"Proceedings of the AAAI Conference","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2019] Kezhen Chen, Irina Rabkina, Matthew McLure, and Kenneth Forbus. Human-like sketch object recognition via analogical learning. Proceedings of the AAAI Conference, 33, 2019.","title":"Human-like sketch object recognition via analogical learning","authors":"Chen","year":"2019","venue":"Proceedings of the AAAI Conference","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2020] Kezhen Chen, Kenneth Forbus, Dedre Gentner, Susan Hespos, and Erie Anderson. Simulating in- fant visual learning by comparison: an initial model. Pro- ceedings of the 42nd Annual Conference of the Cognitive Science Society, 2020.","title":"Simulating infant visual learning by comparison: an initial model","authors":"Chen","year":"2020","venue":"Proceedings of the 42nd Annual Conference of the Cognitive Science Society","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Crouse et al., 2018] Maxwell Crouse, Clifton McFate, and Kenneth Forbus. Learning from unannotated qa pairs to analogically disambiguate and answer questions. Proceed- ings of the AAAI Conference, 32, 2018.","title":"Learning from unannotated qa pairs to analogically disambiguate and answer questions","authors":"Maxwell Crouse, Clifton Mcfate, Kenneth Forbus","year":"2018","venue":"Proceedings of the AAAI Conference","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Eitz et al., 2012] Mathias Eitz, James Hays, , and Marc Alexa. How do humans sketch objects? ACM Transac- tions on graphics, 2012.","title":"How do humans sketch objects?","authors":"Mathias Eitz, James Hays, Marc Alexa","year":"2012","venue":"ACM Transactions on graphics","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Forbus et al., 1995] Kenneth Forbus, Dedre Gentner, and Keith Law. Mac/fac: A model of similarity-based retrieval. Cognitive Science, pages 141-205, 1995.","title":"Mac/fac: A model of similarity-based retrieval","authors":"Kenneth Forbus, Dedre Gentner, Keith Law","year":"1995","venue":"Cognitive Science","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Forbus et al., 2011] Kenneth Forbus, Jeffrey Usher, Andrew Lovett, Kate Lockwood, , and Jon Wetzel. Cogsketch: Sketch understanding for cognitive science research and for education. Cognitive Science, pages 648-666, 2011.","title":"Cogsketch: Sketch understanding for cognitive science research and for education","authors":"Kenneth Forbus, Jeffrey Usher, Andrew Lovett, Kate Lockwood, Jon Wetzel","year":"2011","venue":"Cognitive Science","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Forbus et al., 2017] Kenneth Forbus, Ronald Ferguson, An- drew Lovett, and Dedre Gentner. Extending sme to handle large-scale cognitive modeling. Cognitive Science, 2017.","title":"Extending sme to handle large-scale cognitive modeling","authors":"Kenneth Forbus, Ronald Ferguson, Andrew Lovett, Dedre Gentner","year":"2017","venue":"Cognitive Science","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gentner, 1983] Dedre Gentner. Structure-mapping: A theo- retical framework for analogy. Cognitive Science, 1983.","title":"Structure-mapping: A theoretical framework for analogy","authors":"Dedre Gentner","year":"1983","venue":"Cognitive Science","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hammond and Davis, 2007] Tracy Hammond and Randall Davis. Ladder, a sketching language for user interface de- velopers. ACM SIGGRAPH, 2007.","title":"Ladder, a sketching language for user interface developers","authors":"Tracy Hammond, Randall Davis","year":"2007","venue":"ACM SIGGRAPH","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. Proceedings of IEEE conference on computer vi- sion and pattern recognition, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proceedings of IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jurgen Schmidhuber. Long short-term memory. Neural computation, 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jurgen Schmidhuber","year":"1997","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Krenski and Selinger, 2003] Karol Krenski and Peter Selinger. Potrace. http://potrace.sourceforge.net/, 2003. Accessed: 2003-08-10.","title":"Potrace","authors":"Selinger Krenski, Karol Krenski, Peter Selinger","year":"2003","venue":null,"doi":null,"url":"http://potrace.sourceforge.net/","internal_url":null},{"position":16,"raw":"[Lake et al., 2015] Brenden Lake, Ruslan Salakutdinov, and Joshua Tenenbaum. Human-level concept learning through probabilistic program induction. Proceedings of IEEE, 2015.","title":"Human-level concept learning through probabilistic program induction","authors":"Lake","year":"2015","venue":"Proceedings of IEEE","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[LeCun et al., 1998] Yann LeCun, Leon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of IEEE, 1998.","title":"Gradient-based learning applied to document recognition","authors":"Yann Lecun, Leon Bottou, Yoshua Bengio, Patrick Haffner","year":"1998","venue":"Proceedings of IEEE","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Li et al., 2013] Yi Li, Song Yizhe, , and Shaogang Gong. Sketch recognition by ensemble matching of structured features. BMVC, 2013.","title":"Sketch recognition by ensemble matching of structured features","authors":"Yi Li, Song Yizhe, Shaogang Gong","year":"2013","venue":"BMVC","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lin et al., 2019] Hangyu Lin, Peng Lu, Yanwei Fu, Shao- gang Gong, Xiangyang Xue, and Yugang Jiang. Tc-net for isbir: Triplet classification network for in-stance-level sketch based image retrieval. ACM Multimedia, 2019.","title":"Tc-net for isbir: Triplet classification network for in-stance-level sketch based image retrieval","authors":"Hangyu Lin, Peng Lu, Yanwei Fu, Shaogang Gong, Xiangyang Xue, Yugang Jiang","year":"2019","venue":"ACM Multimedia","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Lin et al., 2020] Hangyu Lin, Yanwei Fu, Xiangyang Xue, and Yugang Jiang. Sketch-bert: Learning sketch bidirec- tional encoder rep-resentation from transformers by self- supervised learning of sketch gestalt. Proceedings of IEEE conference on computer vision and pattern recognition, 2020.","title":"Sketch-bert: Learning sketch bidirectional encoder rep-resentation from transformers by selfsupervised learning of sketch gestalt","authors":"Lin","year":"2020","venue":"Proceedings of IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Lovett and Forbus, 2013] Andrew Lovett and Kenneth For- bus. Modeling spatial ability in mental rotation and pa- per-folding. Proceedings of the Annual Conference of the Cognitive Science Society, 2013.","title":"Modeling spatial ability in mental rotation and paper-folding","authors":"Forbus Lovett","year":"2013","venue":"Proceedings of the Annual Conference of the Cognitive Science Society","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Lovett and Forbus, 2017] Andrew Lovett and Kenneth For- bus. Modeling visual problem solving as analogical rea- soning. Journal of Psychological Review, 2017.","title":"Andrew Lovett and Kenneth Forbus. Modeling visual problem solving as analogical reasoning","authors":"Forbus Lovett","year":"2017","venue":"Journal of Psychological Review","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[McLure et al., 2015a] Matthew McLure, Scott Friedman, and Kenneth Forbus. Extending analogical generalization with near-misses. Proceedings of the AAAI Conference, 2015.","title":"Extending analogical generalization with near-misses","authors":"Mclure","year":"2015","venue":"Proceedings of the AAAI Conference","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[McLure et al., 2015b] Matthew McLure, Subu Kan- daswamy, and Kenneth Forbus. Finding textures in sketches using planar ising models. 28th International Workshop on Qualitative Reasoning, 2015.","title":"Finding textures in sketches using planar ising models","authors":"Mclure","year":"2015","venue":"28th International Workshop on Qualitative Reasoning","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Negroponte, 1973] N. Negroponte. Recent advances in sketch recognition. Proceedings of the National Computer Conference and Exposition, 1973.","title":"Recent advances in sketch recognition","authors":"N Negroponte","year":"1973","venue":"Proceedings of the National Computer Conference and Exposition","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Randell et al., 1992] David Randell, Zhan Cui, and An- thony Cohn. A spatial logic based on regions and con- nection. KR, 1992.","title":"A spatial logic based on regions and connection","authors":"David Randell, Zhan Cui, Anthony Cohn","year":"1992","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Seddati et al., 2015] Omar Seddati, Stephane Dupont, and Said Mahmoudi. Deepsketch: Deep convolutional neural networks for sketch recognition and similarity search. 13th international workshop on content-based multime-dia in- dexing, 2015.","title":"Deepsketch: Deep convolutional neural networks for sketch recognition and similarity search. 13th international workshop on content-based multime-dia indexing","authors":"Omar Seddati, Stephane Dupont, Said Mahmoudi","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Yu et al., 2016] Qian Yu, Feng Liu, Yizhe Song, Tao Xiang, Timothy Hospedales, and Chen-Change Loy. Sketch me that show. Proceedings of IEEE conference on computer vision and pattern recognition, 2016.","title":"Sketch me that show","authors":"Qian Yu, Feng Liu, Yizhe Song, Tao Xiang, Timothy Hospedales, Chen-Change Loy","year":"2016","venue":"Proceedings of IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhang and Suen, 1984] Tongjie Y. Zhang and Ching Y. Suen. A fast parallel algorithm for thinning digital pat- terns. Communications of the ACM, 1984.","title":"A fast parallel algorithm for thinning digital patterns","authors":"Y Zhang And Suen ; ] Tongjie, Ching Y Zhang, Suen","year":"1984","venue":"Communications of the ACM","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}