{"type":"ScholarlyArticle","title":"Relation-enhanced DETR for Component Detection in Graphic Design Reverse Engineering","authors":[{"name":"Xixuan Hao","slug":"xixuan-hao","given":"Xixuan","family":"Hao","affiliation":"The University of Hong Kong"},{"name":"Danqing Huang","slug":"danqing-huang","given":"Danqing","family":"Huang","affiliation":"Microsoft Research"},{"name":"Jieru Lin","slug":"jieru-lin","given":"Jieru","family":"Lin","affiliation":"Harbin Institute of Technology"},{"name":"Chin-Yew Lin","slug":"chin-yew-lin","given":"Chin-Yew","family":"Lin","affiliation":"Microsoft Research"}],"abstract":"It is a common practice for designers to create digital prototypes from a mock-up/screenshot. Reverse engineering graphic design by detecting its components (e.g., text, icon, button) helps expedite this process. This paper first conducts a statistical analysis to emphasize the importance of relations in graphic layouts, which further motivates us to incorporate relation modeling into component detection.  Built on the current state-of-the-art DETR (DEtection TRansformer), we introduce a learnable relation matrix to model class correlations. Specifically, the matrix will be added in the DETR decoder to update the query-to-query self-attention. Experiment results on three public datasets show that our approach achieves better performance than several strong baselines. We further visualize the learnt relation matrix and observe some reasonable patterns. Moreover, we show an application of component detection where we leverage the detection outputs as augmented training data for layout generation, which achieves promising results.","doi":"10.24963/ijcai.2023/532","url":"https://ijcai.org/proceedings/2023/532","pdf":"https://ijcai.org/proceedings/2023/0532.pdf","markdown":"https://ijcai.org/proceedings/2023/532.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/532","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Multidisciplinary Topics and Applications","track":"Main Track","number":532,"pages":{"first":4785,"last":4793},"keywords":["Multidisciplinary Topics and Applications: MDA: Arts and creativity","Computer Vision: CV: Recognition (object detection, categorization)","Computer Vision: CV: Applications"],"videos":[],"references":[{"position":1,"raw":"[Bi et al., 2022] Hengyue Bi, Canhui Xu, Cao Shi, Guozhu Liu, Yuteng Li, Honghong Zhang, and Jing Qu. Srrv: A novel document object detector based on spatial-related relation and vision. IEEE Transactions on Multimedia, 2022.","title":"Srrv: A novel document object detector based on spatial-related relation and vision","authors":"Bi","year":"2022","venue":"IEEE Transactions on Multimedia","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Binmakhashen and Mahmoud, 2019] Galal M. Bin- makhashen and Sabri A. Mahmoud. Document layout analysis: A comprehensive survey. ACM Comput. Surv., 52(6), oct 2019.","title":"Document layout analysis: A comprehensive survey","authors":"M Galal, Binmakhashen, A Sabri, Mahmoud","year":"2019","venue":"ACM Comput. Surv","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Carion et al., 2020] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pages 213-229. Springer, 2020.","title":"End-to-end object detection with transformers","authors":"Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, Sergey Zagoruyko","year":"2020","venue":"European conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2019] Zhao-Min Chen, Xiu-Shen Wei, Peng Wang, and Yanwen Guo. Multi-label image recognition with graph convolutional networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5177-5186, 2019.","title":"Multi-label image recognition with graph convolutional networks","authors":"Xiu-Shen Zhao-Min Chen, Peng Wei, Yanwen Wang, Guo","year":"2019","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Deka et al., 2017] Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. Rico: A mobile app dataset for building data-driven design applications. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology, pages 845-854, 2017.","title":"Rico: A mobile app dataset for building data-driven design applications","authors":"Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, Ranjitha Kumar","year":"2017","venue":"Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dixon and Fogarty, 2010] Morgan Dixon and James Foga- rty. Prefab: implementing advanced behaviors using pixel- based reverse engineering of interface structure. In Pro- ceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 1525-1534, 2010.","title":"Morgan Dixon and James Fogarty. Prefab: implementing advanced behaviors using pixelbased reverse engineering of interface structure","authors":"Fogarty Dixon","year":"2010","venue":"Proceedings of the SIGCHI Conference on Human Factors in Computing Systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gao et al., 2021] Peng Gao, Minghang Zheng, Xiaogang Wang, Jifeng Dai, and Hongsheng Li. Fast convergence of detr with spatially modulated co-attention. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3621-3630, 2021.","title":"Fast convergence of detr with spatially modulated co-attention","authors":"Peng Gao, Minghang Zheng, Xiaogang Wang, Jifeng Dai, Hongsheng Li","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Girshick, 2015] Ross Girshick. Fast r-cnn. In Proceedings of the IEEE International Conference on Computer Vision, pages 1440-1448, 2015.","title":"Fast r-cnn","authors":"Ross Girshick","year":"2015","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gupta et al., 2021] Kamal Gupta, Justin Lazarow, Alessan- dro Achille, Larry S Davis, Vijay Mahadevan, and Abhi- nav Shrivastava. Layouttransformer: Layout generation and completion with self-attention. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1004-1014, 2021.","title":"Layouttransformer: Layout generation and completion with self-attention","authors":"Kamal Gupta, Justin Lazarow, Alessandro Achille, Larry S Davis, Vijay Mahadevan, Abhinav Shrivastava","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Harley et al., 2015] Adam W Harley, Alex Ufkes, and Kon- stantinos G Derpanis. Evaluation of deep convolutional nets for document image classification and retrieval. In In- ternational Conference on Document Analysis and Recog- nition (ICDAR), 2015.","title":"Evaluation of deep convolutional nets for document image classification and retrieval","authors":"Adam W Harley, Alex Ufkes, Konstantinos G Derpanis","year":"2015","venue":"International Conference on Document Analysis and Recognition (ICDAR)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[He et al., 2015] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Spatial pyramid pooling in deep con- volutional networks for visual recognition. IEEE Trans- actions on Pattern Analysis and Machine Intelligence, 37(9):1904-1916, 2015.","title":"Spatial pyramid pooling in deep convolutional networks for visual recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2015","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[He et al., 2017] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE International Conference on Computer Vision, pages 2961-2969, 2017.","title":"Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn","authors":"He","year":"2017","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Ionescu et al., 2020] B. Ionescu, H Müller, R Péteri, A. B. Abacha, V. Datla, S. A. Hasan, D. Demner-Fushman, S. Kozlovski, V. Liauchuk, and Y. D. Cid. Overview of the imageclef 2020: Multimedia retrieval in medical, lifel- ogging, nature, and internet applications. Springer, Cham, 2020.","title":"Overview of the imageclef 2020: Multimedia retrieval in medical, lifelogging, nature, and internet applications","authors":"B Ionescu, Müller, A B Péteri, V Abacha, S A Datla, D Hasan, S Demner-Fushman, V Kozlovski, Y D Liauchuk, Cid","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Jang et al., 2016] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144, 2016.","title":"Categorical reparameterization with gumbel-softmax","authors":"Eric Jang, Shixiang Gu, Ben Poole","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Jiang et al., 2018] Chenhan Jiang, Hang Xu, Xiaodan Liang, and Liang Lin. Hybrid knowledge routed modules for large-scale object detection. Advances in Neural Infor- mation Processing Systems, 31, 2018.","title":"Hybrid knowledge routed modules for large-scale object detection","authors":"Chenhan Jiang, Hang Xu, Xiaodan Liang, Liang Lin","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lin et al., 2014] Tsung-Yi Lin, Michael Maire, Serge Be- longie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Com- mon objects in context. In European Conference on Com- puter Vision, pages 740-755. Springer, 2014.","title":"Microsoft coco: Common objects in context","authors":"Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, C Lawrence, Zitnick","year":"2014","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lin et al., 2017] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense ob- ject detection. In Proceedings of the IEEE International Conference on Computer Vision, pages 2980-2988, 2017.","title":"Focal loss for dense object detection","authors":"Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, Piotr Dollár","year":"2017","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Liu et al., 2022] Shilong Liu, Feng Li, Hao Zhang, Xiao Yang, Xianbiao Qi, Hang Su, Jun Zhu, and Lei Zhang. Dab-detr: Dynamic anchor boxes are better queries for detr. arXiv preprint arXiv:2201.12329, 2022.","title":"Dab-detr: Dynamic anchor boxes are better queries for detr","authors":"Shilong Liu, Feng Li, Hao Zhang, Xiao Yang, Xianbiao Qi, Hang Su, Jun Zhu, Lei Zhang","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Loshchilov and Hutter, 2017] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.","title":"Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization","authors":"Hutter Loshchilov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Ma et al., 2021] Tianlong Ma, Xingjiao Wu, Xin Li, Xi- angcheng Du, Zhao Zhou, Liang Xue, and Cheng Jin. Document layout analysis with aesthetic-guided image augmentation. arXiv preprint arXiv:2111.13809, 2021.","title":"Document layout analysis with aesthetic-guided image augmentation","authors":"Tianlong Ma, Xingjiao Wu, Xin Li, Xiangcheng Du, Zhao Zhou, Liang Xue, Cheng Jin","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Manandhar et al., 2020] Dipu Manandhar, Dan Ruta, and John Collomosse. Learning structural similarity of user in- terface layouts using graph networks. In European Confer- ence on Computer Vision, pages 730-746. Springer, 2020.","title":"Learning structural similarity of user interface layouts using graph networks","authors":"Dipu Manandhar, Dan Ruta, John Collomosse","year":"2020","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Manandhar et al., 2021] Dipu Manandhar, Hailin Jin, and John Collomosse. Magic layouts: Structural prior for com- ponent detection in user interface designs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 15809-15818, 2021.","title":"Magic layouts: Structural prior for component detection in user interface designs","authors":"Dipu Manandhar, Jin Hailin, John Collomosse","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Meng et al., 2021] Depu Meng, Xiaokang Chen, Zejia Fan, Gang Zeng, Houqiang Li, Yuhui Yuan, Lei Sun, and Jing- dong Wang. Conditional detr for fast training convergence. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3651-3660, 2021.","title":"Conditional detr for fast training convergence","authors":"Depu Meng, Xiaokang Chen, Zejia Fan, Gang Zeng, Houqiang Li, Yuhui Yuan, Lei Sun, Jingdong Wang","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Moran et al., 2018] Kevin Moran, Carlos Bernal-Cárdenas, Michael Curcio, Richard Bonett, and Denys Poshyvanyk. Machine learning-based prototyping of graphical user in- terfaces for mobile apps. IEEE Transactions on Software Engineering, 46(2):196-221, 2018.","title":"Machine learning-based prototyping of graphical user interfaces for mobile apps","authors":"Kevin Moran, Carlos Bernal-Cárdenas, Michael Curcio, Richard Bonett, Denys Poshyvanyk","year":"2018","venue":"IEEE Transactions on Software Engineering","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Nguyen and Csallner, 2015] Tuan Anh Nguyen and Christoph Csallner. Reverse engineering mobile ap- plication user interfaces with remaui (t). In 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 248-259. IEEE, 2015.","title":"Reverse engineering mobile application user interfaces with remaui (t)","authors":"Anh Tuan, Christoph Nguyen, Csallner","year":"2015","venue":"30th IEEE/ACM International Conference on Automated Software Engineering (ASE)","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Ren et al., 2015] Shaoqing Ren, Kaiming He, Ross Gir- shick, and Jian Sun. Faster r-cnn: Towards real-time ob- ject detection with region proposal networks. Advances in neural information processing systems, 28, 2015.","title":"Faster r-cnn: Towards real-time object detection with region proposal networks","authors":"Ren","year":"2015","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Shi et al., 2022] Danqing Shi, Weiwei Cui, Danqing Huang, Haidong Zhang, and Nan Cao. Reverse-engineering in- formation presentations: Recovering hierarchical group- ing from layouts of visual elements. arXiv preprint arXiv:2201.05194, 2022.","title":"Reverse-engineering information presentations: Recovering hierarchical grouping from layouts of visual elements","authors":"Shi","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[van den Oord et al., 2017] Aaron van den Oord, Oriol Vinyals, and koray kavukcuoglu. Neural discrete repre- sentation learning. In Advances in Neural Information Processing Systems, volume 30.","title":"Aaron van den Oord, Oriol Vinyals, and koray kavukcuoglu. Neural discrete representation learning","authors":"Oord Van Den","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"Curran Associates, Inc., 2017.","title":null,"authors":"Curran Associates, Inc","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wang et al., 2022a] Chien-Yao Wang, Alexey Bochkovskiy, and Hong-Yuan Mark Liao. Yolov7: Trainable bag-of- freebies sets new state-of-the-art for real-time object de- tectors. arXiv preprint arXiv:2207.02696, 2022.","title":"Yolov7: Trainable bag-offreebies sets new state-of-the-art for real-time object detectors","authors":"Wang","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wang et al., 2022b] Yingming Wang, Xiangyu Zhang, Tong Yang, and Jian Sun. Anchor detr: Query design for transformer-based detector. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pages 2567-2575, 2022.","title":"Anchor detr: Query design for transformer-based detector","authors":"Wang","year":"2022","venue":"Proceedings of the AAAI conference on artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Xu et al., 2019] Hang Xu, Chenhan Jiang, Xiaodan Liang, and Zhenguo Li. Spatial-aware graph relation network for large-scale object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9298-9307, 2019.","title":"Spatial-aware graph relation network for large-scale object detection","authors":"Hang Xu, Chenhan Jiang, Xiaodan Liang, Zhenguo Li","year":"2019","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Yamaguchi, 2021] Kota Yamaguchi. Canvasvae: Learning to generate vector graphic documents. In Proceedings of the IEEE/CVF International Conference on Computer Vi- sion, pages 5481-5489, 2021.","title":"Canvasvae: Learning to generate vector graphic documents","authors":"Kota Yamaguchi, Yamaguchi","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Yao et al., 2021] Zhuyu Yao, Jiangbo Ai, Boxun Li, and Chi Zhang. Efficient detr: improving end-to-end object de- tector with dense prior. arXiv preprint arXiv:2104.01318, 2021.","title":"Efficient detr: improving end-to-end object detector with dense prior","authors":"Yao","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Yeh et al., 2009] Tom Yeh, Tsung-Hsiang Chang, and Robert C Miller. Sikuli: using gui screenshots for search and automation. In Proceedings of the 22nd annual ACM symposium on User interface software and technology, pages 183-192, 2009.","title":"Sikuli: using gui screenshots for search and automation","authors":"Tom Yeh, Tsung-Hsiang Chang, Robert C Miller","year":"2009","venue":"Proceedings of the 22nd annual ACM symposium on User interface software and technology","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Zhang et al., 2022] Gongjie Zhang, Zhipeng Luo, Yingchen Yu, Kaiwen Cui, and Shijian Lu. Accelerating detr con- vergence via semantic-aligned matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 949-958, 2022.","title":"Accelerating detr convergence via semantic-aligned matching","authors":"Zhang","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Zhao et al., 2021] Jiawei Zhao, Ke Yan, Yifan Zhao, Xi- aowei Guo, Feiyue Huang, and Jia Li. Transformer-based dual relation graph for multi-label image recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 163-172, 2021.","title":"Transformer-based dual relation graph for multi-label image recognition","authors":"Jiawei Zhao, Ke Yan, Yifan Zhao, Xiaowei Guo, Feiyue Huang, Jia Li","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Zhong et al., 2019] Xu Zhong, Jianbin Tang, and Anto- nio Jimeno Yepes. Publaynet: largest dataset ever for doc- ument layout analysis. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 1015-1022. IEEE, 2019.","title":"Publaynet: largest dataset ever for document layout analysis","authors":"Xu Zhong, Jianbin Tang, Antonio Jimeno, Yepes","year":"2019","venue":"2019 International Conference on Document Analysis and Recognition (ICDAR)","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Zhu et al., 2021] X. Zhu, W. Su, L. Lu, B. Li, X. Wang, and J. Dai. Deformable detr: Deformable transformers for end- to-end object detection. In International Conference on Learning Representations, 2021.","title":"Deformable detr: Deformable transformers for endto-end object detection","authors":"Zhu","year":"2021","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}