{"type":"ScholarlyArticle","title":"RMGN: A Regional Mask Guided Network for Parser-free Virtual Try-on","authors":[{"name":"Chao Lin","slug":"chao-lin","given":"Chao","family":"Lin","affiliation":""},{"name":"Zhao Li","slug":"zhao-li","given":"Zhao","family":"Li","affiliation":"Zhejiang University; Link2Do Technology Ltd"},{"name":"Sheng Zhou","slug":"sheng-zhou","given":"Sheng","family":"Zhou","affiliation":"Zhejiang University"},{"name":"Shichang Hu","slug":"shichang-hu","given":"Shichang","family":"Hu","affiliation":""},{"name":"Jialun Zhang","slug":"jialun-zhang","given":"Jialun","family":"Zhang","affiliation":""},{"name":"Linhao Luo","slug":"linhao-luo","given":"Linhao","family":"Luo","affiliation":"Monash University"},{"name":"Jiarun Zhang","slug":"jiarun-zhang","given":"Jiarun","family":"Zhang","affiliation":""},{"name":"Longtao Huang","slug":"longtao-huang","given":"Longtao","family":"Huang","affiliation":""},{"name":"Yuan He","slug":"yuan-he","given":"Yuan","family":"He","affiliation":""}],"abstract":"Virtual try-on (VTON) aims at fitting target clothes to reference person images, which is widely adopted in e-commerce. Existing VTON approaches can be narrowly categorized into Parser-Based (PB) and Parser-Free (PF) by whether relying on the parser information to mask the persons’clothes and synthesize try-on images. Although abandoning parser information has improved the applicability of PF methods, the ability of detail synthesizing has also been sacrificed. As a result, the distraction from original cloth may persist in synthesized images, especially in complicated postures and high resolution applications. To address the aforementioned issue, we propose a novel PF method named Regional Mask Guided Network (RMGN). More specifically, a regional mask is proposed to explicitly fuse the features of target clothes and reference persons so that the persisted distraction can be eliminated. A posture awareness loss and a multi-level feature extractor are further proposed to handle the complicated postures and synthesize high resolution images. Extensive experiments demonstrate that our proposed RMGN outperforms both state-of-the-art PB and PF methods. Ablation studies further verify the effectiveness of modules in RMGN. Code is available at https://github.com/jokerlc/RMGN-VITON.","doi":"10.24963/ijcai.2022/161","url":"https://ijcai.org/proceedings/2022/161","pdf":"https://ijcai.org/proceedings/2022/0161.pdf","markdown":"https://ijcai.org/proceedings/2022/161.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/161","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":161,"pages":{"first":1151,"last":1158},"keywords":["Computer Vision: Applications","Computer Vision: Neural generative models, auto encoders, GANs","Computer Vision: Segmentation"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986657","length_sec":92,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984800","length_sec":284,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Choi et al., 2021] Seunghwan Choi, Sunghyun Park, Min- soo Lee, and Jaegul Choo. Viton-hd: High-resolution virtual try-on via misalignment-aware normalization. In CVPR, pages 14131-14140, 2021.","title":"Viton-hd: High-resolution virtual try-on via misalignment-aware normalization","authors":"Choi","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Dong et al., 2019] Haoye Dong, Xiaodan Liang, Xiaohui Shen, Bochao Wang, Hanjiang Lai, Jia Zhu, Zhiting Hu, and Jian Yin. Towards multi-pose guided virtual try-on network. In CVPR, pages 9026-9035, 2019.","title":"Towards multi-pose guided virtual try-on network","authors":"Dong","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Ge et al., 2021] Yuying Ge, Yibing Song, Ruimao Zhang, Chongjian Ge, Wei Liu, and Ping Luo. Parser-free vir- tual try-on via distilling appearance flows. In CVPR, pages 8485-8493, 2021.","title":"Parser-free virtual try-on via distilling appearance flows","authors":"Yuying Ge, Yibing Song, Ruimao Zhang, Chongjian Ge, Wei Liu, Ping Luo","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Gong et al., 2017] Ke Gong, Xiaodan Liang, Dongyu Zhang, Xiaohui Shen, and Liang Lin. Look into per- son: Self-supervised structure-sensitive learning and a new benchmark for human parsing. In ICCV, pages 932-940, 2017.","title":"Look into person: Self-supervised structure-sensitive learning and a new benchmark for human parsing","authors":"Ke Gong, Xiaodan Liang, Dongyu Zhang, Xiaohui Shen, Liang Lin","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Gu et al., 2019] Shuyang Gu, Jianmin Bao, Hao Yang, Dong Chen, Fang Wen, and Lu Yuan. Mask-guided por- trait editing with conditional gans. In CVPR, pages 3436- 3445, 2019.","title":"Mask-guided portrait editing with conditional gans","authors":"Shuyang Gu, Jianmin Bao, Hao Yang, Dong Chen, Fang Wen, Lu Yuan","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Han et al., 2018] Xintong Han, Zuxuan Wu, Zhe Wu, Ruichi Yu, and Larry S Davis. Viton: An image-based virtual try-on network. In Proceedings of the IEEE con- ference on computer vision and pattern recognition, pages 7543-7552, 2018.","title":"Viton: An image-based virtual try-on network","authors":"Han","year":"2018","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Heusel et al., 2017] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochre- iter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.","title":"Gans trained by a two time-scale update rule converge to a local nash equilibrium","authors":"Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, Sepp Hochreiter","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Issenhuth et al., 2020] Thibaut Issenhuth, Jérémie Mary, and Clément Calauzenes. Do not mask what you do not need to mask: a parser-free virtual try-on. In ECCV, pages 619-635. Springer, 2020.","title":"Do not mask what you do not need to mask: a parser-free virtual try-on","authors":"Thibaut Issenhuth, Jérémie Mary, Clément Calauzenes","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Johnson et al., 2016] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European conference on computer vision, pages 694-711. Springer, 2016.","title":"Perceptual losses for real-time style transfer and super-resolution","authors":"Justin Johnson, Alexandre Alahi, Li Fei-Fei","year":"2016","venue":"European conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lee et al., 2020] Cheng-Han Lee, Ziwei Liu, Lingyun Wu, and Ping Luo. Maskgan: Towards diverse and interac- tive facial image manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5549-5558, 2020.","title":"Maskgan: Towards diverse and interactive facial image manipulation","authors":"Ziwei Cheng-Han Lee, Lingyun Liu, Ping Wu, Luo","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lewis et al., 2021] Kathleen M Lewis, Srivatsan Varadhara- jan, and Ira Kemelmacher-Shlizerman. Tryongan: Body- aware try-on via layered interpolation. ACM Transactions on Graphics (TOG), 40(4):1-10, 2021.","title":"Tryongan: Bodyaware try-on via layered interpolation","authors":"Kathleen M Lewis, Srivatsan Varadharajan, Ira Kemelmacher-Shlizerman","year":"2021","venue":"ACM Transactions on Graphics (TOG)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2019] Lingzhi Li, Jianmin Bao, Hao Yang, Dong Chen, and Fang Wen. Faceshifter: Towards high fi- delity and occlusion aware face swapping. arXiv preprint arXiv:1912.13457, 2019.","title":"Faceshifter: Towards high fidelity and occlusion aware face swapping","authors":"Li","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Minar et al., 2020] Matiur Rahman Minar, TT Tuan, H Ahn, P Rosin, and YK Lai. Cp-vton+: Clothing shape and texture preserving image-based virtual try-on. In CVPR Workshops, 2020.","title":"Cp-vton+: Clothing shape and texture preserving image-based virtual try-on","authors":"Minar","year":"2020","venue":"CVPR Workshops","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Naha et al., 2020] Shujon Naha, Qingyang Xiao, Prianka Banik, Md Alimoor Reza, and David J Crandall. Pose- guided knowledge transfer for object part segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 906- 907, 2020.","title":"Poseguided knowledge transfer for object part segmentation","authors":"Naha","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Park et al., 2019] Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu. Semantic image synthesis with spatially-adaptive normalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2337-2346, 2019.","title":"Semantic image synthesis with spatially-adaptive normalization","authors":"Park","year":"2019","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Sun et al., 2014] Deqing Sun, Stefan Roth, and Michael J Black. A quantitative analysis of current practices in op- tical flow estimation and the principles behind them. In- ternational Journal of Computer Vision, 106(2):115-137, 2014.","title":"A quantitative analysis of current practices in optical flow estimation and the principles behind them","authors":"Deqing Sun, Stefan Roth, Michael J Black","year":"2014","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Tang et al., 2020] Hao Tang, Dan Xu, Yan Yan, Philip HS Torr, and Nicu Sebe. Local class-specific and global image-level generative adversarial networks for semantic- guided scene generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7870-7879, 2020.","title":"Local class-specific and global image-level generative adversarial networks for semanticguided scene generation","authors":"Hao Tang, Dan Xu, Yan Yan, Philip Hs Torr, Nicu Sebe","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ulyanov et al., 2016] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The miss- ing ingredient for fast stylization. arXiv preprint arXiv:1607.08022, 2016.","title":"Instance normalization: The missing ingredient for fast stylization","authors":"Dmitry Ulyanov, Andrea Vedaldi, Victor Lempitsky","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang et al., 2018] Bochao Wang, Huabin Zheng, Xiaodan Liang, Yimin Chen, Liang Lin, and Meng Yang. To- ward characteristic-preserving image-based virtual try-on network. In Proceedings of the European Conference on Computer Vision (ECCV), pages 589-604, 2018.","title":"Toward characteristic-preserving image-based virtual try-on network","authors":"Wang","year":"2018","venue":"Proceedings of the European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Yang et al., 2020] Han Yang, Ruimao Zhang, Xiaobao Guo, Wei Liu, Wangmeng Zuo, and Ping Luo. Towards photo- realistic virtual try-on by adaptively generating-preserving image content. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 7850-7859, 2020.","title":"Towards photorealistic virtual try-on by adaptively generating-preserving image content","authors":"Han Yang, Ruimao Zhang, Xiaobao Guo, Wei Liu, Wangmeng Zuo, Ping Luo","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zeiler and Fergus, 2014] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In ECCV, pages 818-833. Springer, 2014.","title":"Visualizing and understanding convolutional networks","authors":"D Matthew, Rob Zeiler, Fergus","year":"2014","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhu et al., 2020] Peihao Zhu, Rameen Abdal, Yipeng Qin, and Peter Wonka. Sean: Image synthesis with seman- tic region-adaptive normalization. In CVPR, pages 5104- 5113, 2020.","title":"Sean: Image synthesis with semantic region-adaptive normalization","authors":"Zhu","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}