{"type":"ScholarlyArticle","title":"Composition-aware Graphic Layout GAN for Visual-Textual Presentation Designs","authors":[{"name":"Min Zhou","slug":"min-zhou","given":"Min","family":"Zhou","affiliation":"Zhejiang University"},{"name":"Chenchen Xu","slug":"chenchen-xu","given":"Chenchen","family":"Xu","affiliation":"Zhejiang University"},{"name":"Ye Ma","slug":"ye-ma","given":"Ye","family":"Ma","affiliation":"Zhejiang University"},{"name":"Tiezheng Ge","slug":"tiezheng-ge","given":"Tiezheng","family":"Ge","affiliation":"Zhejiang University"},{"name":"Yuning Jiang","slug":"yuning-jiang","given":"Yuning","family":"Jiang","affiliation":"Zhejiang University"},{"name":"Weiwei Xu","slug":"weiwei-xu","given":"Weiwei","family":"Xu","affiliation":"Zhejiang University"}],"abstract":"In this paper, we study the graphic layout generation problem of producing high-quality visual-textual presentation designs for given images. We note that image compositions, which contain not only global semantics but also spatial information, would largely affect layout results. Hence, we propose a deep generative model, dubbed as composition-aware graphic layout GAN (CGL-GAN), to synthesize layouts based on the global and spatial visual contents of input images. To obtain training images from images that already contain manually designed graphic layout data, previous work suggests masking design elements (e.g., texts and embellishments) as model inputs, which inevitably leaves hint of the ground truth. We study the misalignment between the training inputs (with hint masks) and test inputs (without masks), and design a novel domain alignment module (DAM) to narrow this gap. For training, we built a large-scale layout dataset which consists of 60,548 advertising posters with annotated layout information. To evaluate the generated layouts, we propose three novel metrics according to aesthetic intuitions. Through both quantitative and qualitative evaluations, we demonstrate that the proposed model can synthesize high-quality graphic layouts according to image compositions. The data and code will be available at https://github.com/minzhouGithub/CGL-GAN.","doi":"10.24963/ijcai.2022/692","url":"https://ijcai.org/proceedings/2022/692","pdf":"https://ijcai.org/proceedings/2022/0692.pdf","markdown":"https://ijcai.org/proceedings/2022/692.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/692","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Research Papers","track":"AI and Arts","number":692,"pages":{"first":4995,"last":5001},"keywords":["Methods and resources: Machine learning, deep learning, neural models, reinforcement learning","Application domains: Images and visual arts"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987166","length_sec":83,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985309","length_sec":619,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Arroyo et al., 2021] Diego Martín Arroyo, Janis Postels, and Federico Tombari. Variational transformer networks for layout generation. In CVPR, pages 13642-13652, 2021.","title":"Variational transformer networks for layout generation","authors":"Arroyo","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cao et al., 2012] Ying Cao, Antoni B. Chan, and Rynson W. H. Lau. Automatic stylistic manga layout. ACM Trans. Graph., 31(6):141:1-141:10, 2012.","title":"Automatic stylistic manga layout","authors":"Ying Cao, Antoni B Chan, Rynson W H Lau","year":"2012","venue":"ACM Trans. Graph","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Carion et al., 2020] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, pages 213-229, 2020.","title":"End-to-end object detection with transformers","authors":"Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, Sergey Zagoruyko","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chefer et al., 2021] Hila Chefer, Shir Gur, and Lior Wolf. Generic attention-model explainability for interpreting bi-modal and encoder-decoder transformers. CoRR, abs/2103.15679, 2021.","title":"Generic attention-model explainability for interpreting bi-modal and encoder-decoder transformers","authors":"Hila Chefer, Shir Gur, Lior Wolf","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Jacobs et al., 2003] Charles E. Jacobs, Wilmot Li, Evan Schrier, David Bargeron, and David Salesin. Adap- tive grid-based document layout. ACM Trans. Graph., 22(3):838-847, 2003.","title":"Adaptive grid-based document layout","authors":"Charles E Jacobs, Wilmot Li, Evan Schrier, David Bargeron, David Salesin","year":"2003","venue":"ACM Trans. Graph","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Jyothi et al., 2019] Akash Abdu Jyothi, Thibaut Durand, Ji- awei He, Leonid Sigal, and Greg Mori. Layoutvae: Stochastic scene layout generation from a label set. pages 9894-9903. ICCV, 2019.","title":"Layoutvae: Stochastic scene layout generation from a label set","authors":"Abdu Akash, Thibaut Jyothi, Jiawei Durand, Leonid He, Greg Sigal, Mori","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kikuchi et al., 2021] Kotaro Kikuchi, Edgar Simo-Serra, Mayu Otani, and Kota Yamaguchi. Constrained graphic layout generation via latent optimization. pages 88-96. ACM Multimedia Conference, 2021.","title":"Constrained graphic layout generation via latent optimization","authors":"Kotaro Kikuchi, Edgar Simo-Serra, Mayu Otani, Kota Yamaguchi","year":"2021","venue":"ACM Multimedia Conference","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kingma and Ba, 2015] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, ICLR, San Diego, CA, USA, May 7-9, 2015, 2015.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kumar et al., 2011] Ranjitha Kumar, Jerry O. Talton, Salman Ahmad, and Scott R. Klemmer. Bricolage: example-based retargeting for web design. In Proceed- ings of the International Conference on Human Factors in Computing Systems, pages 2197-2206, 2011.","title":"Bricolage: example-based retargeting for web design","authors":"Ranjitha Kumar, Jerry O Talton, Salman Ahmad, Scott R Klemmer","year":"2011","venue":"Proceedings of the International Conference on Human Factors in Computing Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lee et al., 2019] Hsin-Ying Lee, Weilong Yang, Lu Jiang, Madison Le, Irfan Essa, Haifeng Gong, and Ming-Hsuan Yang. Neural design network: Graphic layout generation with constraints. CoRR, abs/1912.09421, 2019.","title":"Neural design network: Graphic layout generation with constraints","authors":"Hsin-Ying Lee, Weilong Yang, Lu Jiang, Madison Le, Irfan Essa, Haifeng Gong, Ming-Hsuan Yang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lee et al., 2020] Hsin-Ying Lee, Lu Jiang, Irfan Essa, Phuong B. Le, Haifeng Gong, Ming-Hsuan Yang, and Weilong Yang. Neural design network: Graphic layout generation with constraints. pages 491-506. ECCV, 2020.","title":"Neural design network: Graphic layout generation with constraints","authors":"Hsin-Ying Lee, Lu Jiang, Irfan Essa, Phuong B Le, Haifeng Gong, Ming-Hsuan Yang, Weilong Yang","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2019] Jianan Li, Jimei Yang, Aaron Hertzmann, Jianming Zhang, and Tingfa Xu. Layoutgan: Generat- ing graphic layouts with wireframe discriminators. CoRR, abs/1901.06767, 2019.","title":"Layoutgan: Generating graphic layouts with wireframe discriminators","authors":"Jianan Li, Jimei Yang, Aaron Hertzmann, Jianming Zhang, Tingfa Xu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2021] Jianan Li, Jimei Yang, Jianming Zhang, Chang Liu, Christina Wang, and Tingfa Xu. Attribute- conditioned layout gan for automatic graphic design. IEEE Transactions on Visualization and Computer Graphics, 27(10):4039-4048, 2021.","title":"Attributeconditioned layout gan for automatic graphic design","authors":"Jianan Li, Jimei Yang, Jianming Zhang, Chang Liu, Christina Wang, Tingfa Xu","year":"2021","venue":"IEEE Transactions on Visualization and Computer Graphics","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lim and Ye, 2017] Jae Hyun Lim and Jong Chul Ye. Geo- metric GAN. CoRR, abs/1705.02894, 2017.","title":"Geometric GAN","authors":"Jae, Hyun Lim, Jong Chul, Ye","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lin et al., 2017] Tsung-Yi Lin, Piotr Dollár, Ross B. Gir- shick, Kaiming He, Bharath Hariharan, and Serge J. Be- longie. Feature pyramid networks for object detection. In CVPR, pages 936-944, 2017.","title":"Belongie. Feature pyramid networks for object detection","authors":"Tsung-Yi Lin, Piotr Dollár, Ross B Girshick, Kaiming He, Bharath Hariharan, Serge J","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[O'Donovan et al., 2014] Peter O'Donovan, Aseem Agar- wala, and Aaron Hertzmann. Learning layouts for single- pagegraphic designs. IEEE Trans. Vis. Comput. Graph., 20(8):1200-1213, 2014.","title":"Learning layouts for singlepagegraphic designs","authors":"O'donovan","year":"2014","venue":"IEEE Trans. Vis. Comput. Graph","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Radford et al., 2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervi- sion. In ICML, volume 139, pages 8748-8763, 2021.","title":"Learning transferable visual models from natural language supervision","authors":"Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, Ilya Sutskever","year":"2021","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Simonyan and Zisserman, 2015] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Suvorov et al., 2022] Roman Suvorov, Elizaveta Lo- gacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Aleksei Silvestrov, Naejin Kong, Harshith Goka, Kiwoong Park, and Victor Lempitsky. Resolution- robust large mask inpainting with fourier convolutions. In IEEE/CVF Winter Conference on Applications of Com- puter Vision, WACV 2022, Waikoloa, HI, USA, January 3-8, 2022, pages 3172-3182. IEEE, 2022.","title":"Resolutionrobust large mask inpainting with fourier convolutions","authors":"Suvorov","year":"2022","venue":"IEEE/CVF Winter Conference on Applications of Computer Vision, WACV 2022","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2019] Bo Wang, Quan Chen, Min Zhou, Zhiqiang Zhang, Xiaogang Jin, and Kun Gai. Progres- sive feature polishing network for salient object detection. CoRR, abs/1911.05942, 2019.","title":"Progressive feature polishing network for salient object detection","authors":"Bo Wang, Quan Chen, Min Zhou, Zhiqiang Zhang, Xiaogang Jin, Kun Gai","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Yang et al., 2016] Xuyong Yang, Tao Mei, Ying-Qing Xu, Yong Rui, and Shipeng Li. Automatic generation of visual-textual presentation layout. ACM Transactions on Multimedia Computing Communications and Applications (TOMM), 12, March 2016.","title":"Automatic generation of visual-textual presentation layout","authors":"Xuyong Yang, Tao Mei, Ying-Qing Xu, Yong Rui, Shipeng Li","year":"2016","venue":"ACM Transactions on Multimedia Computing Communications and Applications (TOMM)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yang et al., 2021] Cheng-Fu Yang, Wan-Cyuan Fan, Fu-En Yang, and Yu-Chiang Frank Wang. Layouttransformer: Scene layout generation with conceptual and spatial diver- sity. pages 3732-3741. CVPR, 2021.","title":"Layouttransformer: Scene layout generation with conceptual and spatial diversity","authors":"Cheng-Fu Yang, Wan-Cyuan Fan, Fu-En, Yu-Chiang Yang, Wang Frank","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zheng et al., 2019] Xinru Zheng, Xiaotian Qiao, Ying Cao, and Rynson W. H. Lau. Content-aware generative mod- eling of graphic design layouts. ACM Trans. Graph., 38(4):133:1-133:15, 2019.","title":"Content-aware generative modeling of graphic design layouts","authors":"Zheng","year":"2019","venue":"ACM Trans. Graph","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}