{"type":"ScholarlyArticle","title":"Image-to-Image Translation with Multi-Path Consistency Regularization","authors":[{"name":"Jianxin Lin","slug":"jianxin-lin","given":"Jianxin","family":"Lin","affiliation":"CAS Key Laboratory of Technology in Geo-spatial Information Processing and Application System , University of Science and Technology of China"},{"name":"Yingce Xia","slug":"yingce-xia","given":"Yingce","family":"Xia","affiliation":"Microsoft Research Asia"},{"name":"Yijun Wang","slug":"yijun-wang","given":"Yijun","family":"Wang","affiliation":"University of Science and Technology of China"},{"name":"Tao Qin","slug":"tao-qin","given":"Tao","family":"Qin","affiliation":"Microsoft Research Asia"},{"name":"Zhibo Chen","slug":"zhibo-chen","given":"Zhibo","family":"Chen","affiliation":"CAS Key Laboratory of Technology in Geo-spatial Information Processing and Application System , University of Science and Technology of China"}],"abstract":"Image translation across different domains has attracted much attention in both machine learning and computer vision communities. Taking the translation from a source domain to a target domain as an example, existing algorithms mainly rely on two kinds of loss for training: One is the discrimination loss, which is used to differentiate images generated by the models and natural images; the other is the reconstruction loss, which measures the difference between an original image and the reconstructed version. In this work, we introduce a new kind of loss, multi-path consistency loss, which evaluates the differences between direct translation from source domain to target domain and indirect translation from source domain to an auxiliary domain to target domain, to regularize training. For multi-domain translation (at least, three) which focuses on building translation models between any two domains, at each training iteration, we randomly select three domains, set them respectively as the source, auxiliary and target domains, build the multi-path consistency loss and optimize the network. For two-domain translation, we need to introduce an additional auxiliary domain and construct the multi-path consistency loss. We conduct various experiments to demonstrate the effectiveness of our proposed methods, including face-to-face translation, paint-to-photo translation, and de-raining/de-noising translation.","doi":"10.24963/ijcai.2019/413","url":"https://ijcai.org/proceedings/2019/413","pdf":"https://ijcai.org/proceedings/2019/0413.pdf","markdown":"https://ijcai.org/proceedings/2019/413.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/413","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning A-L","track":"Main track","number":413,"pages":{"first":2980,"last":2986},"keywords":["Machine Learning: Learning Generative Models","Computer Vision: Computer Vision"],"videos":[],"references":[{"position":1,"raw":"[Arjovsky et al., 2017] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International Conference on Machine Learning, pages 214-223, 2017.","title":"Wasserstein generative adversarial networks","authors":"Arjovsky","year":"2017","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2016] Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Inter- pretable representation learning by information maximizing gen- erative adversarial nets. In Advances in Neural Information Pro- cessing Systems, pages 2172-2180, 2016.","title":"Infogan: Interpretable representation learning by information maximizing generative adversarial nets","authors":"Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, Pieter Abbeel","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Choi et al., 2018] Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Woo Ha, Sunghun Kim, and Jaegul Choo. Stargan: Uni- fied generative adversarial networks for multi-domain image-to- image translation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.","title":"Stargan: Unified generative adversarial networks for multi-domain image-toimage translation","authors":"Choi","year":"2018","venue":"The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Donahue et al., 2016] Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell. Adversarial feature learning. arXiv preprint arX- iv:1605.09782, 2016.","title":"Adversarial feature learning","authors":"Jeff Donahue, Philipp Krähenbühl, Trevor Darrell","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dumoulin et al., 2016] Vincent Dumoulin, Ishmael Belghazi, Ben Poole, Alex Lamb, Martin Arjovsky, Olivier Mastropietro, and Aaron Courville. Adversarially learned inference. arXiv preprint arXiv:1606.00704, 2016.","title":"Adversarially learned inference","authors":"Ishmael Vincent Dumoulin, Ben Belghazi, Alex Poole, Martin Lamb, Olivier Arjovsky, Aaron Mastropietro, Courville","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fu et al., 2017] Xueyang Fu, Jiabin Huang, Delu Zeng, Yue Huang, Xinghao Ding, and John Paisley. Removing rain from single images via a deep detail network. In IEEE Conference on Computer Vision and Pattern Recognition, pages 1715-1723, 2017.","title":"Removing rain from single images via a deep detail network","authors":"Xueyang Fu, Jiabin Huang, Delu Zeng, Yue Huang, Xinghao Ding, John Paisley","year":"2017","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gatys et al., 2016] Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2414-2423, 2016.","title":"Image style transfer using convolutional neural networks","authors":"Leon A Gatys, Alexander S Ecker, Matthias Bethge","year":"2016","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Girshick, 2015] Ross Girshick. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 1440- 1448, 2015.","title":"Fast r-cnn","authors":"Ross Girshick","year":"2015","venue":"Proceedings of the IEEE international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Goodfellow et al., 2014] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial net- s. In Advances in neural information processing systems, pages 2672-2680, 2014.","title":"Generative adversarial nets","authors":"Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio","year":"2014","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[He et al., 2016] Di He, Yingce Xia, Tao Qin, Liwei Wang, Neng- hai Yu, Tieyan Liu, and Wei-Ying Ma. Dual learning for machine translation. In Advances in Neural Information Processing Sys- tems, pages 820-828, 2016.","title":"Dual learning for machine translation","authors":"Di He, Yingce Xia, Tao Qin, Liwei Wang, Nenghai Yu, Tieyan Liu, Wei-Ying Ma","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Heusel et al., 2017] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 6626-6637.","title":"Gans trained by a two time-scale update rule converge to a local nash equilibrium","authors":"Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, Sepp Hochreiter","year":null,"venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"Curran Associates, Inc., 2017.","title":null,"authors":"Curran Associates, Inc","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Huang et al., 2017] Xun Huang, Yixuan Li, Omid Poursaeed, John E Hopcroft, and Serge J Belongie. Stacked generative ad- versarial networks. In CVPR, volume 2, page 3, 2017.","title":"Stacked generative adversarial networks","authors":"Xun Huang, Yixuan Li, Omid Poursaeed, John E Hopcroft, Serge J Belongie","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Isola et al., 2017] P. Isola, J. Zhu, T. Zhou, and A. A. Efros. Image- to-image translation with conditional adversarial networks. In 2017 IEEE Conference on Computer Vision and Pattern Recog- nition (CVPR), pages 5967-5976, July 2017.","title":"Imageto-image translation with conditional adversarial networks","authors":"P Isola, J Zhu, T Zhou, A A Efros","year":"2017","venue":"IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kim et al., 2017] Taeksoo Kim, Moonsu Cha, Hyunsoo Kim, Jung Kwon Lee, and Jiwon Kim. Learning to discover cross- domain relations with generative adversarial networks. In Pro- ceedings of the 34th International Conference on Machine Learn- ing, pages 1857-1865, 2017.","title":"Learning to discover crossdomain relations with generative adversarial networks","authors":"Kim","year":"2017","venue":"Proceedings of the 34th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kingma and Ba, 2014] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arX- iv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"Diederik Kingma, Jimmy Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Ledig et al., 2017] C. Ledig, L. Theis, F. Huszár, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang, and W. Shi. Photo-realistic single image super- resolution using a generative adversarial network. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 105-114, July 2017.","title":"Photo-realistic single image superresolution using a generative adversarial network","authors":"C Ledig, L Theis, F Huszár, J Caballero, A Cunningham, A Acosta, A Aitken, A Tejani, J Totz, Z Wang, W Shi","year":"2017","venue":"IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lin et al., 2018a] Jianxin Lin, Yingce Xia, Tao Qin, Zhibo Chen, and Tie-Yan Liu. Conditional image-to-image translation. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018.","title":"Conditional image-to-image translation","authors":"Lin","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lin et al., 2018b] Jianxin Lin, Tiankuang Zhou, and Zhibo Chen. Multi-scale face restoration with sequential gating ensemble net- work. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 7122-7129, 2018.","title":"Multi-scale face restoration with sequential gating ensemble network","authors":"Lin","year":"2018","venue":"Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Liu et al., 2015] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xi- aoou Tang. Deep learning face attributes in the wild. In Pro- ceedings of International Conference on Computer Vision (IC- CV), 2015.","title":"Deep learning face attributes in the wild","authors":"Ziwei Liu, Ping Luo, Xiaogang Wang, Xiaoou Tang","year":"2015","venue":"Proceedings of International Conference on Computer Vision (IC-CV)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Liu et al., 2018] Alexander H Liu, Yen-Cheng Liu, Yu-Ying Yeh, and Yu-Chiang Frank Wang. A unified feature disentangler for multi-domain image translation and manipulation. In Advances in Neural Information Processing Systems, pages 2590-2599, 2018.","title":"A unified feature disentangler for multi-domain image translation and manipulation","authors":"Yen-Cheng Alexander H Liu, Yu-Ying Liu, Yu-Chiang Yeh, Wang Frank","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Mao et al., 2016] Xiaojiao Mao, Chunhua Shen, and Yu-Bin Yang. Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections. In Advances in Neu- ral Information Processing Systems 29, pages 2802-2810, 2016.","title":"Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections","authors":"Xiaojiao Mao, Chunhua Shen, Yu-Bin Yang","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Radford et al., 2015] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep con- volutional generative adversarial networks. arXiv preprint arX- iv:1511.06434, 2015.","title":"Unsupervised representation learning with deep convolutional generative adversarial networks","authors":"Radford","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Taigman et al., 2016] Yaniv Taigman, Adam Polyak, and Lior Wolf. Unsupervised cross-domain image generation. arXiv preprint arXiv:1611.02200, 2016.","title":"Unsupervised cross-domain image generation","authors":"Yaniv Taigman, Adam Polyak, Lior Wolf","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yang et al., 2017] Wenhan Yang, Robby T Tan, Jiashi Feng, Jiay- ing Liu, Zongming Guo, and Shuicheng Yan. Deep joint rain detection and removal from a single image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1357-1366, 2017.","title":"Deep joint rain detection and removal from a single image","authors":"Wenhan Yang, Robby T Tan, Jiashi Feng, Jiaying Liu, Zongming Guo, Shuicheng Yan","year":"2017","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yi et al., 2017] Zili Yi, Hao Zhang, Ping Tan, and Minglun Gong. Dualgan: Unsupervised dual learning for image-to-image trans- lation. In The IEEE International Conference on Computer Vi- sion (ICCV), Oct 2017.","title":"Dualgan: Unsupervised dual learning for image-to-image translation","authors":"Zili Yi, Hao Zhang, Ping Tan, Minglun Gong","year":"2017","venue":"The IEEE International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhu et al., 2017] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In The IEEE International Conference on Computer Vision (ICCV), Oct 2017.","title":"Unpaired image-to-image translation using cycle-consistent adversarial networks","authors":"Zhu","year":"2017","venue":"The IEEE International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}