{"type":"ScholarlyArticle","title":"Self-Supervised Learning for Enhancing Spatial Awareness in Free-Hand Sketches","authors":[{"name":"Xin Wang","slug":"xin-wang","given":"Xin","family":"Wang","affiliation":"Dept. of Computer Science and Engineering , Shanghai Jiao Tong University , Shanghai 200240 , China"},{"name":"Tengjie Li","slug":"tengjie-li","given":"Tengjie","family":"Li","affiliation":"Dept. of Computer Science and Engineering , Shanghai Jiao Tong University , Shanghai 200240 , China"},{"name":"Sicong Zang","slug":"sicong-zang","given":"Sicong","family":"Zang","affiliation":"Dept. of Computer Science and Engineering , Shanghai Jiao Tong University , Shanghai 200240 , China; School of Computer Science and Technology , Donghua University , Shanghai 201620 , China"},{"name":"Shikui Tu","slug":"shikui-tu","given":"Shikui","family":"Tu","affiliation":"Dept. of Computer Science and Engineering , Shanghai Jiao Tong University , Shanghai 200240 , China"},{"name":"Lei Xu","slug":"lei-xu","given":"Lei","family":"Xu","affiliation":"Dept. of Computer Science and Engineering , Shanghai Jiao Tong University , Shanghai 200240 , China; Guangdong Institute of Intelligence Science and Technology , Zhuhai , Guangdong 519031 , China"}],"abstract":"Free-hand sketch, as a versatile medium of communication, can be viewed as a collection of strokes arranged in a spatial layout to convey a concept. Due to the abstract nature of the sketches, changes in stroke position may make them difficult to recognize. Recently, Graphic sketch representations are effective in representing sketches. However, existing methods overlook the significance of the spatial layout of strokes and the phenomenon of strokes being drawn in the wrong positions is common. Therefore, we developed a self-supervised task to correct stroke placement and investigate the impact of spatial layout on learning sketch representations. For this task, we propose a spatially aware method, named SketchGloc, utilizing multiple graphs for graphic sketch representations. This method utilizes grids for each stroke to describe the spatial layout with other strokes, allowing for the construction of multiple graphs. Unlike other methods that rely on a single graph, this design conveys more detailed spatial layout information and alleviates the impact of misplaced strokes. The experimental results demonstrate that our model outperforms existing methods in both our proposed task and the traditional controllable sketch synthesis task. Additionally, we found that SketchGloc can learn more robust representations under our proposed task setting. The source code is available at https://github.com/CMACH508/SketchGloc.","doi":"10.24963/ijcai.2024/566","url":"https://ijcai.org/proceedings/2024/566","pdf":"https://ijcai.org/proceedings/2024/0566.pdf","markdown":"https://ijcai.org/proceedings/2024/566.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/566","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Machine Learning","track":"Main Track","number":566,"pages":{"first":5117,"last":5125},"keywords":["Computer Vision: CV: Representation learning","Machine Learning: ML: Generative models","Humans and AI: HAI: Applications","Multidisciplinary Topics and Applications: MTA: Arts and creativity"],"videos":[],"references":[{"position":1,"raw":"[Armstrong and Okun, 2020] Melissa J Armstrong and Michael S Okun. Diagnosis and treatment of parkinson disease: a review. Jama, 323(6):548-560, 2020.","title":"Diagnosis and treatment of parkinson disease: a review","authors":"J Melissa, Michael S Armstrong, Okun","year":"2020","venue":"Jama","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bhunia et al., 2021] Ayan Kumar Bhunia, Pinaki Nath Chowdhury, Yongxin Yang, Timothy M Hospedales, Tao Xiang, and Yi-Zhe Song. Vectorization and rasterization: Self-supervised learning for sketch and handwriting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5672-5681, 2021.","title":"Vectorization and rasterization: Self-supervised learning for sketch and handwriting","authors":"Ayan Kumar Bhunia, Pinaki Nath Chowdhury, Yongxin Yang, Timothy M Hospedales, Tao Xiang, Yi-Zhe Song","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chancellor et al., 2014] Bree Chancellor, Angel Duncan, and Anjan Chatterjee. Art therapy for alzheimer's dis- ease and other dementias. Journal of Alzheimer's Disease, 39(1):1-11, 2014.","title":"Bree Chancellor, Angel Duncan, and Anjan Chatterjee. Art therapy for alzheimer's disease and other dementias","authors":"Chancellor","year":"2014","venue":"Journal of Alzheimer's Disease","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2017] Yajing Chen, Shikui Tu, Yuqi Yi, and Lei Xu. Sketch-pix2seq: a model to generate sketches of mul- tiple categories. arXiv preprint arXiv:1709.04121, 2017.","title":"Sketch-pix2seq: a model to generate sketches of multiple categories","authors":"Chen","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Doersch et al., 2015] Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learn- ing by context prediction. In Proceedings of the IEEE in- ternational conference on computer vision, pages 1422- 1430, 2015.","title":"Unsupervised visual representation learning by context prediction","authors":"Carl Doersch, Abhinav Gupta, Alexei A Efros","year":"2015","venue":"Proceedings of the IEEE international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Ge et al., 2020] Songwei Ge, Vedanuj Goswami, C Lawrence Zitnick, and Devi Parikh. Creative sketch generation. arXiv preprint arXiv:2011.10039, 2020.","title":"Creative sketch generation","authors":"Vedanuj Songwei Ge, C Goswami, Devi Lawrence Zitnick, Parikh","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gidaris et al., 2018] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In 6th International Confer- ence on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 -May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018.","title":"Unsupervised representation learning by predicting image rotations","authors":"Spyros Gidaris, Praveer Singh, Nikos Komodakis","year":"2018","venue":"6th International Conference on Learning Representations, ICLR 2018","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Girshick, 2015] Ross Girshick. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 1440-1448, 2015.","title":"Fast r-cnn","authors":"Ross Girshick","year":"2015","venue":"Proceedings of the IEEE international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ha and Eck, 2017] David Ha and Douglas Eck. A neu- ral representation of sketch drawings. arXiv preprint arXiv:1704.03477, 2017.","title":"A neural representation of sketch drawings","authors":"Eck Ha, David Ha, Douglas Eck","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ha and Eck, 2018] David Ha and Douglas Eck. A neu- ral representation of sketch drawings. In 6th Interna- tional Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 -May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018.","title":"A neural representation of sketch drawings","authors":"Eck Ha, David Ha, Douglas Eck","year":"2018","venue":"6th International Conference on Learning Representations, ICLR 2018","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kipf and Welling, 2016] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907, 2016.","title":"Semi-supervised classification with graph convolutional networks","authors":"N Thomas, Max Kipf, Welling","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2020] Lei Li, Changqing Zou, Youyi Zheng, Qingkun Su, Hongbo Fu, and Chiew-Lan Tai. Sketch- r2cnn: an rnn-rasterization-cnn architecture for vector sketch recognition. IEEE transactions on visualization and computer graphics, 27(9):3745-3754, 2020.","title":"Sketchr2cnn: an rnn-rasterization-cnn architecture for vector sketch recognition","authors":"Lei Li, Changqing Zou, Youyi Zheng, Qingkun Su, Hongbo Fu, Chiew-Lan Tai","year":"2020","venue":"IEEE transactions on visualization and computer graphics","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2024] Tengjie Li, Sicong Zang, Shikui Tu, and Lei Xu. Lmser-pix2seq: Learning stable sketch representa- tions for sketch healing. Computer Vision and Image Un- derstanding, 240:103931, 2024.","title":"Lmser-pix2seq: Learning stable sketch representations for sketch healing","authors":"Li","year":"2024","venue":"Computer Vision and Image Understanding","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lin et al., 2020] Hangyu Lin, Yanwei Fu, Xiangyang Xue, and Yu-Gang Jiang. Sketch-bert: Learning sketch bidirec- tional encoder representation from transformers by self- supervised learning of sketch gestalt. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 6758-6767, 2020.","title":"Sketch-bert: Learning sketch bidirectional encoder representation from transformers by selfsupervised learning of sketch gestalt","authors":"Lin","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2016] Wei Liu, Dragomir Anguelov, Dumitru Er- han, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C Berg. Ssd: Single shot multibox detector. In Computer Vision-ECCV 2016: 14th European Confer- ence, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part I 14, pages 21-37. Springer, 2016.","title":"Ssd: Single shot multibox detector","authors":"Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, Alexander C Berg","year":"2016","venue":"Computer Vision-ECCV 2016: 14th European Conference","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Pan et al., 2020] Chen Pan, Yafeng Han, and Jiping Lu. De- sign and optimization of lattice structures: A review. Ap- plied Sciences, 10(18):6374, 2020.","title":"Design and optimization of lattice structures: A review","authors":"Chen Pan, Yafeng Han, Jiping Lu","year":"2020","venue":"Applied Sciences","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Pathak et al., 2016] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A. Efros. Con- text encoders: Feature learning by inpainting. In Proceed- ings of the IEEE Conference on Computer Vision and Pat- tern Recognition (CVPR), June 2016.","title":"Context encoders: Feature learning by inpainting","authors":"Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, Alexei A Efros","year":"2016","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Qi et al., 2021] Yonggang Qi, Guoyao Su, Pinaki Nath Chowdhury, Mingkang Li, and Yi-Zhe Song. Sketchlat- tice: Latticed representation for sketch manipulation. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10- 17, 2021, pages 933-941. IEEE, 2021.","title":"Sketchlattice: Latticed representation for sketch manipulation","authors":"Yonggang Qi, Guoyao Su, Pinaki Nath Chowdhury, Mingkang Li, Yi-Zhe Song","year":"2021","venue":"IEEE/CVF International Conference on Computer Vision, ICCV 2021","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Qi et al., 2022] Yonggang Qi, Guoyao Su, Qiang Wang, Jie Yang, Kaiyue Pang, and Yi-Zhe Song. Generative sketch healing. International Journal of Computer Vision, 130(8):2006-2021, 2022.","title":"Generative sketch healing","authors":"Yonggang Qi, Guoyao Su, Qiang Wang, Jie Yang, Kaiyue Pang, Yi-Zhe Song","year":"2022","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Qu et al., 2023] Zhiyu Qu, Tao Xiang, and Yi-Zhe Song. Sketchdreamer: Interactive text-augmented creative sketch ideation. arXiv preprint arXiv:2308.14191, 2023.","title":"Sketchdreamer: Interactive text-augmented creative sketch ideation","authors":"Zhiyu Qu, Tao Xiang, Yi-Zhe Song","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Redmon et al., 2016] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 779-788, 2016.","title":"You only look once: Unified, real-time object detection","authors":"Joseph Redmon, Santosh Divvala, Ross Girshick, Ali Farhadi","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Ribeiro et al., 2020] Leo Sampaio Ferraz Ribeiro, Tu Bui, John Collomosse, and Moacir Ponti. Sketchformer: Transformer-based representation for sketched structure. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14153-14162, 2020.","title":"Sketchformer: Transformer-based representation for sketched structure","authors":"Ribeiro","year":"2020","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Song et al., 2018] Jifei Song, Kaiyue Pang, Yi-Zhe Song, Tao Xiang, and Timothy M Hospedales. Learning to sketch with shortcut cycle consistency. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 801-810, 2018.","title":"Learning to sketch with shortcut cycle consistency","authors":"Song","year":"2018","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Su et al., 2020] Guoyao Su, Yonggang Qi, Kaiyue Pang, Jie Yang, and Yi-Zhe Song. Sketchhealer: A graph-to- sequence network for recreating partial human sketches. In 31st British Machine Vision Conference 2020, BMVC 2020, Virtual Event, UK, September 7-10, 2020. BMVA Press, 2020.","title":"Sketchhealer: A graph-tosequence network for recreating partial human sketches","authors":"Su","year":"2020","venue":"31st British Machine Vision Conference 2020, BMVC 2020, Virtual Event","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Vinker et al., 2022] Yael Vinker, Ehsan Pajouheshgar, Jes- sica Y Bo, Roman Christian Bachmann, Amit Haim Bermano, Daniel Cohen-Or, Amir Zamir, and Ariel Shamir. Clipasso: Semantically-aware object sketching. ACM Transactions on Graphics (TOG), 41(4):1-11, 2022.","title":"Clipasso: Semantically-aware object sketching","authors":"Yael Vinker, Ehsan Pajouheshgar, Jessica Y Bo, Roman Christian Bachmann, Amit Haim Bermano, Daniel Cohen-Or, Amir Zamir, Ariel Shamir","year":"2022","venue":"ACM Transactions on Graphics (TOG)","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Vinker et al., 2023] Yael Vinker, Yuval Alaluf, Daniel Cohen-Or, and Ariel Shamir. Clipascene: Scene sketching with different types and levels of abstraction. In Proceed- ings of the IEEE/CVF International Conference on Com- puter Vision, pages 4146-4156, 2023.","title":"Clipascene: Scene sketching with different types and levels of abstraction","authors":"Yael Vinker, Yuval Alaluf, Daniel Cohen-Or, Ariel Shamir","year":"2023","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Voynov et al., 2023] Andrey Voynov, Kfir Aberman, and Daniel Cohen-Or. Sketch-guided text-to-image diffusion models. In ACM SIGGRAPH 2023 Conference Proceed- ings, pages 1-11, 2023.","title":"Sketch-guided text-to-image diffusion models","authors":"Andrey Voynov, Kfir Aberman, Daniel Cohen-Or","year":"2023","venue":"ACM SIGGRAPH 2023 Conference Proceedings","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wang et al., 2022] Qiang Wang, Haoge Deng, Yonggang Qi, Da Li, and Yi-Zhe Song. Sketchknitter: Vectorized sketch generation with diffusion models. In The Eleventh International Conference on Learning Representations, 2022.","title":"Sketchknitter: Vectorized sketch generation with diffusion models","authors":"Qiang Wang, Haoge Deng, Yonggang Qi, Da Li, Yi-Zhe Song","year":"2022","venue":"The Eleventh International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Xing et al., 2024] Ximing Xing, Chuang Wang, Haitao Zhou, Jing Zhang, Qian Yu, and Dong Xu. Diffsketcher: Text guided vector sketch synthesis through latent diffu- sion models. Advances in Neural Information Processing Systems, 36, 2024.","title":"Diffsketcher: Text guided vector sketch synthesis through latent diffusion models","authors":"Ximing Xing, Chuang Wang, Haitao Zhou, Jing Zhang, Qian Yu, Dong Xu","year":"2024","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Xu et al., 2020a] Peng Xu, Yongye Huang, Tongtong Yuan, Tao Xiang, Timothy M Hospedales, Yi-Zhe Song, and Liang Wang. On learning semantic representations for million-scale free-hand sketches. arXiv preprint arXiv:2007.04101, 2020.","title":"On learning semantic representations for million-scale free-hand sketches","authors":"Xu","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Xu et al., 2020b] Peng Xu, Zeyu Song, Qiyue Yin, Yi-Zhe Song, and Liang Wang. Deep self-supervised representa- tion learning for free-hand sketch. IEEE Transactions on Circuits and Systems for Video Technology, 31(4):1503- 1513, 2020.","title":"Deep self-supervised representation learning for free-hand sketch","authors":"Xu","year":"2020","venue":"IEEE Transactions on Circuits and Systems for Video Technology","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Xu et al., 2022] Peng Xu, Timothy M Hospedales, Qiyue Yin, Yi-Zhe Song, Tao Xiang, and Liang Wang. Deep learning for free-hand sketch: A survey. IEEE transactions on pattern analysis and machine intelligence, 45(1):285- 312, 2022.","title":"Deep learning for free-hand sketch: A survey","authors":"Peng Xu, Timothy M Hospedales, Qiyue Yin, Yi-Zhe Song, Tao Xiang, Liang Wang","year":"2022","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Yu et al., 2017] Qian Yu, Yongxin Yang, Feng Liu, Yi-Zhe Song, Tao Xiang, and Timothy M Hospedales. Sketch-a- net: A deep neural network that beats humans. Interna- tional journal of computer vision, 122:411-425, 2017.","title":"Sketch-anet: A deep neural network that beats humans","authors":"Qian Yu, Yongxin Yang, Feng Liu, Yi-Zhe Song, Tao Xiang, Timothy M Hospedales","year":"2017","venue":"International journal of computer vision","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zang et al., 2021] Sicong Zang, Shikui Tu, and Lei Xu. Controllable stroke-based sketch synthesis from a self- organized latent space. Neural Networks, 137:138-150, 2021.","title":"Controllable stroke-based sketch synthesis from a selforganized latent space","authors":"Zang","year":"2021","venue":"Neural Networks","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zang et al., 2023] Sicong Zang, Shikui Tu, and Lei Xu. Linking sketch patches by learning synonymous proxim- ity for graphic sketch representation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 11096-11103, 2023.","title":"Linking sketch patches by learning synonymous proximity for graphic sketch representation","authors":"Zang","year":"2023","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Zhao et al., 2019] Zhong-Qiu Zhao, Peng Zheng, Shou-tao Xu, and Xindong Wu. Object detection with deep learn- ing: A review. IEEE transactions on neural networks and learning systems, 30(11):3212-3232, 2019.","title":"Object detection with deep learning: A review","authors":"Zhong-Qiu Zhao, Peng Zheng, Shou-Tao Xu, Xindong Wu","year":"2019","venue":"IEEE transactions on neural networks and learning systems","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Zou et al., 2023] Zhengxia Zou, Keyan Chen, Zhenwei Shi, Yuhong Guo, and Jieping Ye. Object detection in 20 years: A survey. Proceedings of the IEEE, 2023.","title":"Object detection in 20 years: A survey","authors":"Zou","year":"2023","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}