Self-Supervised Learning for Enhancing Spatial Awareness in Free-Hand Sketches
Self-Supervised Learning for Enhancing Spatial Awareness in Free-Hand Sketches
Xin Wang, Tengjie Li, Sicong Zang, Shikui Tu, Lei Xu
Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence
Main Track. Pages 5117-5125.
https://doi.org/10.24963/ijcai.2024/566
Free-hand sketch, as a versatile medium of communication, can be viewed as a collection of strokes arranged in a spatial layout to convey a concept. Due to the abstract nature of the sketches, changes in stroke position may make them difficult to recognize. Recently, Graphic sketch representations are effective in representing sketches. However, existing methods overlook the significance of the spatial layout of strokes and the phenomenon of strokes being drawn in the wrong positions is common. Therefore, we developed a self-supervised task to correct stroke placement and investigate the impact of spatial layout on learning sketch representations. For this task, we propose a spatially aware method, named SketchGloc, utilizing multiple graphs for graphic sketch representations. This method utilizes grids for each stroke to describe the spatial layout with other strokes, allowing for the construction of multiple graphs. Unlike other methods that rely on a single graph, this design conveys more detailed spatial layout information and alleviates the impact of misplaced strokes. The experimental results demonstrate that our model outperforms existing methods in both our proposed task and the traditional controllable sketch synthesis task. Additionally, we found that SketchGloc can learn more robust representations under our proposed task setting. The source code is available at https://github.com/CMACH508/SketchGloc.
Keywords:
Machine Learning: ML: Generative models
Computer Vision: CV: Representation learning
Humans and AI: HAI: Applications
Multidisciplinary Topics and Applications: MTA: Arts and creativity