{"type":"ScholarlyArticle","title":"Deep Video Harmonization With Color Mapping Consistency","authors":[{"name":"Xinyuan Lu","slug":"xinyuan-lu","given":"Xinyuan","family":"Lu","affiliation":"MoE Key Lab of Artificial Intelligence , Department of Computer Science and Engineering Shanghai Jiaotong University , Shanghai , China"},{"name":"Shengyuan Huang","slug":"shengyuan-huang","given":"Shengyuan","family":"Huang","affiliation":"MoE Key Lab of Artificial Intelligence , Department of Computer Science and Engineering Shanghai Jiaotong University , Shanghai , China"},{"name":"Li Niu","slug":"li-niu","given":"Li","family":"Niu","affiliation":"MoE Key Lab of Artificial Intelligence , Department of Computer Science and Engineering Shanghai Jiaotong University , Shanghai , China"},{"name":"Wenyan Cong","slug":"wenyan-cong","given":"Wenyan","family":"Cong","affiliation":"MoE Key Lab of Artificial Intelligence , Department of Computer Science and Engineering Shanghai Jiaotong University , Shanghai , China"},{"name":"Liqing Zhang","slug":"liqing-zhang","given":"Liqing","family":"Zhang","affiliation":"MoE Key Lab of Artificial Intelligence , Department of Computer Science and Engineering Shanghai Jiaotong University , Shanghai , China"}],"abstract":"Video harmonization aims to adjust the foreground of a composite video to make it compatible with the background. So far, video harmonization has only received limited attention and there is no public dataset for video harmonization. In this work, we construct a new video harmonization dataset HYouTube by adjusting the foreground of real videos to create synthetic composite videos. Moreover, we consider the temporal consistency in video harmonization task. Unlike previous works which establish the spatial correspondence, we design a novel framework based on the assumption of color mapping consistency, which leverages the color mapping of neighboring frames to refine the current frame. Extensive experiments on our HYouTube dataset prove the effectiveness of our proposed framework. Our dataset and code are available at https://github.com/bcmi/Video-Harmonization-Dataset-HYouTube.","doi":"10.24963/ijcai.2022/172","url":"https://ijcai.org/proceedings/2022/172","pdf":"https://ijcai.org/proceedings/2022/0172.pdf","markdown":"https://ijcai.org/proceedings/2022/172.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/172","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":172,"pages":{"first":1232,"last":1238},"keywords":["Computer Vision: Other","Computer Vision: Video analysis and understanding"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986553","length_sec":65,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984696","length_sec":249,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Bo et al., 2011] Fan Bo, Zhou Fugen, and Han Hong- bin. Medical image enhancement based on modified lut- mapping derivative and multi-scale layer contrast modifi- cation. In International Congress on Image and Signal Processing, 2011.","title":"Medical image enhancement based on modified lutmapping derivative and multi-scale layer contrast modification","authors":"Fan Bo, Han Zhou Fugen, Hongbin","year":"2011","venue":"International Congress on Image and Signal Processing","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cong et al., 2020] Wenyan Cong, Liu Liu Zhixin Ling Weiyuan Li Jianfu Zhang, Li Niu, and Liqing Zhang. Dovenet: Deep image harmonization via domain verifica- tion. In CVPR, 2020.","title":"Dovenet: Deep image harmonization via domain verification","authors":"Wenyan Cong, Liu Liu Zhixin, Ling Weiyuan, Li Jianfu Zhang, Li Niu, Liqing Zhang","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cong et al., 2021] Wenyan Cong, Li Niu, Jianfu Zhang, Jing Liang, and Liqing Zhang. Bargainnet: Background- guided domain translation for image harmonization. In ICME, 2021.","title":"Bargainnet: Backgroundguided domain translation for image harmonization","authors":"Wenyan Cong, Li Niu, Jianfu Zhang, Jing Liang, Liqing Zhang","year":"2021","venue":"ICME","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cong et al., 2022] Wenyan Cong, Xinhao Tao, Li Niu, Jing Liang, Xuesong Gao, Qihao Sun, and Liqing Zhang. High- resolution image harmonization via collaborative dual transformations. CVPR, 2022.","title":"Highresolution image harmonization via collaborative dual transformations","authors":"Wenyan Cong, Xinhao Tao, Li Niu, Jing Liang, Xuesong Gao, Qihao Sun, Liqing Zhang","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cun and Pun, 2020] Xiaodong Cun and Chi-Man Pun. Im- proving the harmony of the composite image by spatial- separated attention module. IEEE Transactions on Image Processing, 29:4759-4771, 2020.","title":"Improving the harmony of the composite image by spatialseparated attention module","authors":"Xiaodong Cun, Chi-Man Pun","year":"2020","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fischl and Schwartz, 1999] Bruce Fischl and Eric L Schwartz. Adaptive nonlocal filtering: a fast alterna- tive to anisotropic diffusion for image enhancement. IEEE Transactions on Pattern Analysis and Machine Intelligence, 21(1):42-48, 1999.","title":"Adaptive nonlocal filtering: a fast alternative to anisotropic diffusion for image enhancement","authors":"Schwartz Fischl, Bruce Fischl, Eric L Schwartz","year":"1999","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Guo et al., 2021] Zonghui Guo, Zhaorui Gu Haiy- ong Zheng, Yufeng Jiang, and Bing Zheng. Intrinsic image harmonization. In CVPR, 2021.","title":"Intrinsic image harmonization","authors":"Guo","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hao et al., 2020] Guoqing Hao, Satoshi Iizuka, and Kazuhiro Fukui. Image harmonization with attention- based deep feature modulation. In BMVC, 2020.","title":"Image harmonization with attentionbased deep feature modulation","authors":"Guoqing Hao, Satoshi Iizuka, Kazuhiro Fukui","year":"2020","venue":"BMVC","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Haozhi et al., 2019] Huang Haozhi, Wei Liu Senzhe Xu, Junxiong Cai, and Shimin Hu. Temporally coherent video harmonization using adversarial networks. IEEE Transac- tions on Image Processing, 29:214-224, 2019.","title":"Temporally coherent video harmonization using adversarial networks","authors":"Huang Haozhi, Wei Liu, Senzhe Xu, Junxiong Cai, Shimin Hu","year":"2019","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kalyan et al., 2010] Sunkavalli Kalyan, Wojciech Matusik Micah K. Johnson, and Hanspeter Pfister. Multi-scale im- age harmonization. Acm Trans Graph, 29(4):1-10, 2010.","title":"Multi-scale image harmonization","authors":"Kalyan","year":"2010","venue":"Acm Trans Graph","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Konstantin et al., 2021] Sofiiuk Konstantin, Polina Popen- ova, and Anton Konushin. Foreground-aware semantic representations for image harmonization. WACV, 2021.","title":"Konushin. Foreground-aware semantic representations for image harmonization","authors":"Konstantin","year":"2021","venue":"Sofiiuk Konstantin","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lalonde and Efros, 2007] Jean-Francois Lalonde and Alexei A Efros. Using color compatibility for assessing image realism. In ICCV, 2007.","title":"Using color compatibility for assessing image realism","authors":"Jean-Francois Lalonde, Alexei A Efros","year":"2007","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Ling et al., 2021] Jun Ling, Han Xue, Li Song, Rong Xie, and Xiao Gu. Region-aware adaptive instance normaliza- tion for image harmonization. In CVPR, 2021.","title":"Region-aware adaptive instance normalization for image harmonization","authors":"Ling","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2020] Yifan Liu, Chunhua Shen, Changqian Yu, and Jingdong Wang. Efficient semantic video segmenta- tion with per-frame inference. In ECCV, 2020.","title":"Efficient semantic video segmentation with per-frame inference","authors":"Yifan Liu, Chunhua Shen, Changqian Yu, Jingdong Wang","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mese Murat, 2001] Palghat P. Vaidyanathan. Mese Murat. Look-up table (lut) method for inverse halftoning. IEEE Transactions on Image Processing, 10(10):1566-1578, 2001.","title":"Look-up table (lut) method for inverse halftoning","authors":"P Palghat, Vaidyanathan, Murat","year":"2001","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Michael et al., 2006] Elad Michael, Boaz Matalon, and Michael Zibulevsky. Image denoising with shrinkage and redundant representations. In CVPR, 2006.","title":"Image denoising with shrinkage and redundant representations","authors":"Elad Michael, Boaz Matalon, Michael Zibulevsky","year":"2006","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Phillip et al., 2016] Isola Phillip, Tinghui Zhou Jun- Yan Zhu, and Alexei A. Efros. Image-to-image translation with conditional adversarial networks. In CVPR, 2016.","title":"Image-to-image translation with conditional adversarial networks","authors":"Isola Phillip, Tinghui Zhou Jun-Yan Zhu, Alexei A Efros","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Reinhard et al., 2001] Erik Reinhard, Bruce Gooch Michael Ashikhmin, and Peter Shirley. Color trans- fer between images. IEEE Computer Graphics and Applications, 21(5):34-41, 2001.","title":"Color transfer between images","authors":"Reinhard","year":"2001","venue":"IEEE Computer Graphics and Applications","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Shu et al., 2015] Fujita Shu, Makoto Kimura Nor- ishige Fukushima, and Yutaka Ishibashi. Randomized redundant dct: efficient denoising by using random subsampling of dct patches. In Siggraph Asia Technical Briefs, 2015.","title":"Randomized redundant dct: efficient denoising by using random subsampling of dct patches","authors":"Fujita Shu, Makoto Kimura Norishige, Yutaka Fukushima, Ishibashi","year":"2015","venue":"Siggraph Asia Technical Briefs","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Su et al., 2012] Xue Su, Julie Dorsey Aseem Agarwala, and Holly Rushmeier. Rushmeier h.: Understanding and im- proving the realism of image composites. In Acm Trans Graph, volume 31, pages 1-10, 2012.","title":"Rushmeier h.: Understanding and improving the realism of image composites","authors":"Xue Su, Julie Dorsey Aseem Agarwala, Holly Rushmeier","year":"2012","venue":"Acm Trans Graph","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sun et al., 2021] Yanan Sun, Guanzhi Wang, Qiao Gu, Chi- Keung Tang, and Yu-Wing Tai. Deep video matting via spatio-temporal alignment and aggregation. In CVPR, 2021.","title":"Deep video matting via spatio-temporal alignment and aggregation","authors":"Yanan Sun, Guanzhi Wang, Qiao Gu, Chi-Keung Tang, Yu-Wing Tai","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Tianfan et al., 2019] Xue Tianfan, Donglai Wei Ba- ian Chen, Jiajun Wu, and William T. Freeman. Video enhancement with task-oriented flow. International Journal of Computer Vision, 127(8):1106-1125, 2019.","title":"Video enhancement with task-oriented flow","authors":"Xue Tianfan, Donglai Wei Baian Chen, Jiajun Wu, William T Freeman","year":"2019","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Xu et al., 2018] Ning Xu, Linjie Yang, Yuchen Fan, Dingcheng Yue, Yuchen Liang, Jianchao Yang, and Thomas Huang. Youtube-vos: A large-scale video object segmentation benchmark. arXiv preprint arXiv:1809.03327, 2018.","title":"Youtube-vos: A large-scale video object segmentation benchmark","authors":"Ning Xu, Linjie Yang, Yuchen Fan, Dingcheng Yue, Yuchen Liang, Jianchao Yang, Thomas Huang","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yi-Hsuan et al., 2017] Tsai Yi-Hsuan, Kalyan Sunkavalli Xin Lu Xiaohui Shen, Zhe Lin, and Ming-Hsuan Yang. Deep image harmonization. In CVPR, 2017.","title":"Deep image harmonization","authors":"Tsai Yi-Hsuan, Kalyan Sunkavalli, Xin Lu Xiaohui Shen, Zhe Lin, Ming-Hsuan Yang","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhu et al., 2015] Jun-Yan Zhu, Eli Shechtman Philipp Kra- henbuhl, and Alexei A. Efros. Learning a discriminative model for the perception of realism in composite images. In ICCV, 2015.","title":"Learning a discriminative model for the perception of realism in composite images","authors":"Zhu","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}