{"type":"ScholarlyArticle","title":"Real-Time Portrait Stylization on the Edge","authors":[{"name":"Yanyu Li","slug":"yanyu-li","given":"Yanyu","family":"Li","affiliation":"Northeastern University"},{"name":"Xuan Shen","slug":"xuan-shen","given":"Xuan","family":"Shen","affiliation":"Northeastern University"},{"name":"Geng Yuan","slug":"geng-yuan","given":"Geng","family":"Yuan","affiliation":"Northeastern University"},{"name":"Jiexiong Guan","slug":"jiexiong-guan","given":"Jiexiong","family":"Guan","affiliation":"College of William & Mary"},{"name":"Wei Niu","slug":"wei-niu","given":"Wei","family":"Niu","affiliation":"College of William & Mary"},{"name":"Hao Tang","slug":"hao-tang","given":"Hao","family":"Tang","affiliation":"CVL , ETH Zurich"},{"name":"Bin Ren","slug":"bin-ren","given":"Bin","family":"Ren","affiliation":"College of William & Mary"},{"name":"Yanzhi Wang","slug":"yanzhi-wang","given":"Yanzhi","family":"Wang","affiliation":"Northeastern University"}],"abstract":"In this work we demonstrate real-time portrait stylization, specifically, translating self-portrait into cartoon or anime style on mobile devices. \r\nWe propose a latency-driven differentiable architecture search method, maintaining realistic generative quality. \r\nWith our framework, we obtain 10× computation reduction on the generative model and achieve real-time video stylization on off-the-shelf smartphone using mobile GPUs.","doi":"10.24963/ijcai.2022/856","url":"https://ijcai.org/proceedings/2022/856","pdf":"https://ijcai.org/proceedings/2022/0856.pdf","markdown":"https://ijcai.org/proceedings/2022/856.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/856","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Demo Track","track":"Demo Track","number":856,"pages":{"first":5928,"last":5931},"keywords":["Machine Learning: Automated Machine Learning","Constraint Satisfaction and Optimization: Applications","Computer Vision: Neural generative models, auto encoders, GANs"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987216","length_sec":60,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985359","length_sec":359,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Andersson and Arvidsson, 2020] Filip Andersson and Si- mon Arvidsson. Generative adversarial networks for photo to hayao miyazaki style cartoons. arXiv preprint arXiv:2005.07702, 2020.","title":"Generative adversarial networks for photo to hayao miyazaki style cartoons","authors":"Arvidsson Andersson","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bengio et al., 2013] Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432, 2013.","title":"Estimating or propagating gradients through stochastic neurons for conditional computation","authors":"Yoshua Bengio, Nicholas Léonard, Aaron Courville","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chang et al., 2020] Sung-En Chang, Yanyu Li, Mengshu Sun, Runbin Shi, Hayden K-H So, Xuehai Qian, Yanzhi Wang, and Xue Lin. Mix and match: A novel fpga-centric deep neural network quantization framework. arXiv preprint arXiv:2012.04240, 2020.","title":"Mix and match: A novel fpga-centric deep neural network quantization framework","authors":"Sung-En Chang, Yanyu Li, Mengshu Sun, Runbin Shi, K-H Hayden, Xuehai So, Yanzhi Qian, Xue Wang, Lin","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2021] Xuxi Chen, Zhenyu Zhang, Yongduo Sui, and Tianlong Chen. Gans can play lottery tickets too. arXiv preprint arXiv:2106.00134, 2021.","title":"Gans can play lottery tickets too","authors":"Chen","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chong and Forsyth, 2021] Min Jin Chong and David Forsyth. Gans n'roses: Stable, controllable, diverse image to image translation (works for videos too!). arXiv preprint arXiv:2106.06561, 2021.","title":"Gans n'roses: Stable, controllable, diverse image to image translation","authors":"Chong, Forsyth, Min, Jin Chong, David Forsyth","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fu et al., 2020] Yonggan Fu, Wuyang Chen, Haotao Wang, Haoran Li, Yingyan Lin, and Zhangyang Wang. Autogan- distiller: Searching to compress generative adversarial net- works. In ICML, 2020.","title":"Autogandistiller: Searching to compress generative adversarial networks","authors":"Fu","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Guan et al., 2020] Yushuo Guan, Ning Liu, Pengyu Zhao, Zhengping Che, Kaigui Bian, Yanzhi Wang, and Jian Tang. Dais: Automatic channel pruning via differ- entiable annealing indicator search. arXiv preprint arXiv:2011.02166, 2020.","title":"Dais: Automatic channel pruning via differentiable annealing indicator search","authors":"Yushuo Guan, Ning Liu, Pengyu Zhao, Zhengping Che, Kaigui Bian, Yanzhi Wang, Jian Tang","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Han et al., 2015] Song Han, Jeff Pool, et al. Learning both weights and connections for efficient neural network. In NeurIPS, 2015.","title":"Learning both weights and connections for efficient neural network","authors":"Han","year":"2015","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[He et al., 2017] Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural net- works. In ICCV, 2017.","title":"Channel pruning for accelerating very deep neural networks","authors":"Yihui He, Xiangyu Zhang, Jian Sun","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Huang et al., 2018] Xun Huang, Ming-Yu Liu, Serge Be- longie, and Jan Kautz. Multimodal unsupervised image- to-image translation. In ECCV, 2018.","title":"Multimodal unsupervised imageto-image translation","authors":"Xun Huang, Ming-Yu Liu, Serge Belongie, Jan Kautz","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Isola et al., 2017] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with con- ditional adversarial networks. In CVPR, 2017.","title":"Image-to-image translation with conditional adversarial networks","authors":"Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, Alexei A Efros","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Jin et al., 2021] Qing Jin, Jian Ren, Oliver J Woodford, Jiazhuo Wang, Geng Yuan, Yanzhi Wang, and Sergey Tulyakov. Teachers do more than teach: Compressing image-to-image models. In CVPR, 2021.","title":"Teachers do more than teach: Compressing image-to-image models","authors":"Qing Jin, Jian Ren, J Oliver, Jiazhuo Woodford, Geng Wang, Yanzhi Yuan, Sergey Wang, Tulyakov","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Karras et al., 2020] Tero Karras, Samuli Laine, Miika Ait- tala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. An- alyzing and improving the image quality of StyleGAN. In CVPR, 2020.","title":"Analyzing and improving the image quality of StyleGAN","authors":"Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, Timo Aila","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kim et al., 2020] Junho Kim, Minjae Kim, Hyeonwoo Kang, and Kwang Hee Lee. U-gat-it: Unsupervised gen- erative attentional networks with adaptive layer-instance normalization for image-to-image translation. In ICLR, 2020.","title":"U-gat-it: Unsupervised generative attentional networks with adaptive layer-instance normalization for image-to-image translation","authors":"Kim","year":"2020","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lee et al., 2018] Hsin-Ying Lee, Hung-Yu Tseng, Jia-Bin Huang, Maneesh Kumar Singh, and Ming-Hsuan Yang. Diverse image-to-image translation via disentangled rep- resentations. In ECCV, 2018.","title":"Diverse image-to-image translation via disentangled representations","authors":"Hsin-Ying Lee, Hung-Yu Tseng, Jia-Bin Huang, Maneesh Kumar Singh, Ming-Hsuan Yang","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li et al., 2020] Muyang Li, Ji Lin, Yaoyao Ding, Zhijian Liu, Jun-Yan Zhu, and Song Han. Gan compression: Ef- ficient architectures for interactive conditional gans. In CVPR, 2020.","title":"Gan compression: Efficient architectures for interactive conditional gans","authors":"Muyang Li, Ji Lin, Yaoyao Ding, Zhijian Liu, Jun-Yan Zhu, Song Han","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Li et al., 2021] Bing Li, Yuanlue Zhu, Yitong Wang, Chia- Wen Lin, Bernard Ghanem, and Linlin Shen. Anigan: Style-guided generative adversarial networks for unsuper- vised anime face generation. IEEE TMM, 2021.","title":"Anigan: Style-guided generative adversarial networks for unsupervised anime face generation","authors":"Bing Li, Yuanlue Zhu, Yitong Wang, Chia-Wen Lin, Bernard Ghanem, Linlin Shen","year":"2021","venue":"IEEE TMM","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Liu et al., 2017] Ming-Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised image-to-image translation net- works. NeurIPS, 2017.","title":"Unsupervised image-to-image translation networks","authors":"Ming-Yu Liu, Thomas Breuel, Jan Kautz","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Liu et al., 2019] Ming-Yu Liu, Xun Huang, Arun Mallya, Tero Karras, Timo Aila, Jaakko Lehtinen, and Jan Kautz. Few-shot unsupervised image-to-image translation. In ICCV, 2019.","title":"Few-shot unsupervised image-to-image translation","authors":"Ming-Yu Liu, Xun Huang, Arun Mallya, Tero Karras, Timo Aila, Jaakko Lehtinen, Jan Kautz","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Park et al., 2020] Taesung Park, Alexei A. Efros, Richard Zhang, and Jun-Yan Zhu. Contrastive learning for un- paired image-to-image translation. In ECCV, 2020.","title":"Contrastive learning for unpaired image-to-image translation","authors":"Park","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Tang et al., 2019] Hao Tang, Dan Xu, Nicu Sebe, Yanzhi Wang, Jason J Corso, and Yan Yan. Multi-channel at- tention selection gan with cascaded semantic guidance for cross-view image translation. In CVPR, 2019.","title":"Multi-channel attention selection gan with cascaded semantic guidance for cross-view image translation","authors":"Hao Tang, Dan Xu, Nicu Sebe, Yanzhi Wang, Jason J Corso, Yan Yan","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Tang et al., 2021] Hao Tang, Hong Liu, Dan Xu, Philip HS Torr, and Nicu Sebe. Attentiongan: Unpaired image-to- image translation using attention-guided generative adver- sarial networks. IEEE TNNLS, 2021.","title":"Attentiongan: Unpaired image-toimage translation using attention-guided generative adversarial networks","authors":"Hao Tang, Hong Liu, Dan Xu, Philip Hs Torr, Nicu Sebe","year":"2021","venue":"IEEE TNNLS","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang et al., 2020] Haotao Wang, Shupeng Gui, Haichuan Yang, Ji Liu, and Zhangyang Wang. Gan slimming: All- in-one gan compression by a unified optimization frame- work. In ECCV, 2020.","title":"Gan slimming: Allin-one gan compression by a unified optimization framework","authors":"Wang","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yu et al., 2017] Xiyu Yu, Tongliang Liu, Xinchao Wang, and Dacheng Tao. On compressing deep models by low rank and sparse decomposition. In CVPR, 2017.","title":"On compressing deep models by low rank and sparse decomposition","authors":"Xiyu Yu, Tongliang Liu, Xinchao Wang, Dacheng Tao","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhong et al., 2018] Zhao Zhong, Junjie Yan, Wei Wu, Jing Shao, and Cheng-Lin Liu. Practical block-wise neural net- work architecture generation. In CVPR, 2018.","title":"Practical block-wise neural network architecture generation","authors":"Zhao Zhong, Junjie Yan, Wei Wu, Jing Shao, Cheng-Lin Liu","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhu et al., 2017] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image transla- tion using cycle-consistent adversarial networks. In ICCV, 2017.","title":"Unpaired image-to-image translation using cycle-consistent adversarial networks","authors":"Zhu","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zoph and Le, 2017] Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning. In ICLR, 2017.","title":"Neural architecture search with reinforcement learning","authors":"Le ; Zoph, Barret Zoph, Quoc V Le","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}