{"type":"ScholarlyArticle","title":"Temporal Adaptive Alignment Network for Deep Video Inpainting","authors":[{"name":"Ruixin Liu","slug":"ruixin-liu","given":"Ruixin","family":"Liu","affiliation":"Communication and Information Security Lab , Shenzhen Graduate School , Peking University"},{"name":"Zhenyu Weng","slug":"zhenyu-weng","given":"Zhenyu","family":"Weng","affiliation":"Communication and Information Security Lab , Shenzhen Graduate School , Peking University"},{"name":"Yuesheng Zhu","slug":"yuesheng-zhu","given":"Yuesheng","family":"Zhu","affiliation":"Communication and Information Security Lab , Shenzhen Graduate School , Peking University"},{"name":"Bairong Li","slug":"bairong-li","given":"Bairong","family":"Li","affiliation":"Communication and Information Security Lab , Shenzhen Graduate School , Peking University"}],"abstract":"Video inpainting aims to synthesize visually pleasant and temporally consistent content in missing regions of video. Due to a variety of motions across different frames, it is highly challenging to utilize effective temporal information to recover videos. Existing deep learning based methods usually estimate optical flow to align frames and thereby exploit useful information between frames. However, these methods tend to generate artifacts once the estimated optical flow is inaccurate. To alleviate above problem, we propose a novel end-to-end Temporal Adaptive Alignment Network(TAAN) for video inpainting. The TAAN aligns reference frames with target frame via implicit motion estimation at a feature level and then reconstruct target frame by taking the aggregated aligned reference frame features as input. In the proposed network, a Temporal Adaptive Alignment (TAA) module based on deformable convolutions is designed to perform temporal alignment in a local, dense and adaptive manner. Both quantitative and qualitative evaluation results show that our method significantly outperforms existing deep learning based methods.","doi":"10.24963/ijcai.2020/129","url":"https://ijcai.org/proceedings/2020/129","pdf":"https://ijcai.org/proceedings/2020/0129.pdf","markdown":"https://ijcai.org/proceedings/2020/129.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/129","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Computer Vision","track":"Main track","number":129,"pages":{"first":927,"last":933},"keywords":["Computer Vision: 2D and 3D Computer Vision","Machine Learning Applications: Applications of Unsupervised Learning","Machine Learning: Deep Learning: Convolutional networks"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24450/file_24450.m3u8","length_sec":299,"provider":"hls"}],"references":[{"position":1,"raw":"[Barnes et al., 2009] Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B Goldman. Patchmatch: A randomized correspondence algorithm for structural image editing. In ACM Transactions on Graphics (ToG), volume 28, page 24, 2009.","title":"Patchmatch: A randomized correspondence algorithm for structural image editing","authors":"Connelly Barnes, Eli Shechtman, Adam Finkelstein, Dan B Goldman","year":"2009","venue":"ACM Transactions on Graphics (ToG)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chang et al., 2019] Ya-Liang Chang, Zhe Yu Liu, Kuan- Ying Lee, and Winston Hsu. Free-form video inpainting with 3d gated convolution and temporal patchgan. In Pro- ceedings of the IEEE International Conference on Com- puter Vision, pages 9066-9075, 2019.","title":"Free-form video inpainting with 3d gated convolution and temporal patchgan","authors":"Ya-Liang Chang, Zhe, Yu Liu, Kuan-Ying Lee, Winston Hsu","year":"2019","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Dai et al., 2017] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. De- formable convolutional networks. In Proceedings of the IEEE International Conference on Computer Vision, pages 764-773, 2017.","title":"Deformable convolutional networks","authors":"Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, Yichen Wei","year":"2017","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Granados et al., 2012] Miguel Granados, Kwang In Kim, James Tompkin, Jan Kautz, and Christian Theobalt. Back- ground inpainting for videos with dynamic objects and a free-moving camera. In European Conference on Com- puter Vision, pages 682-695. Springer, 2012.","title":"Background inpainting for videos with dynamic objects and a free-moving camera","authors":"Miguel Granados, In Kwang, James Kim, Jan Tompkin, Christian Kautz, Theobalt","year":"2012","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[He et al., 2015] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpass- ing human-level performance on imagenet classification. In Proceedings of the IEEE International Conference on Computer Vision, pages 1026-1034, 2015.","title":"Delving deep into rectifiers: Surpassing human-level performance on imagenet classification","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2015","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Huang et al., 2016] Jia-Bin Huang, Sing Bing Kang, Naren- dra Ahuja, and Johannes Kopf. Temporally coherent com- pletion of dynamic video. ACM Transactions on Graphics (TOG), 35(6):196, 2016.","title":"Temporally coherent completion of dynamic video","authors":"Jia-Bin Huang, Bing Sing, Narendra Kang, Johannes Ahuja, Kopf","year":"2016","venue":"ACM Transactions on Graphics (TOG)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Iizuka et al., 2017] Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. Globally and locally consistent im- age completion. ACM Transactions on Graphics (ToG), 36(4):107, 2017.","title":"Globally and locally consistent image completion","authors":"Satoshi Iizuka, Edgar Simo-Serra, Hiroshi Ishikawa","year":"2017","venue":"ACM Transactions on Graphics (ToG)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kim et al., 2019] Dahun Kim, Sanghyun Woo, Joon-Young Lee, and In So Kweon. Deep video inpainting. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5792-5801, 2019.","title":"Deep video inpainting","authors":"Kim","year":"2019","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Lee et al., 2019] Sungho Lee, Seoung Wug Oh, DaeYeun Won, and Seon Joo Kim. Copy-and-paste networks for deep video inpainting. In Proceedings of the IEEE Interna- tional Conference on Computer Vision, pages 4413-4421, 2019.","title":"Copy-and-paste networks for deep video inpainting","authors":"Sungho Lee, Seoung Wug Oh, Daeyeun Won, Seon Joo Kim","year":"2019","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Liu et al., 2018] Guilin Liu, Fitsum A Reda, Kevin J Shih, Ting-Chun Wang, Andrew Tao, and Bryan Catanzaro. Im- age inpainting for irregular holes using partial convolu- tions. In Proceedings of the European Conference on Computer Vision, pages 85-100, 2018.","title":"Image inpainting for irregular holes using partial convolutions","authors":"Liu","year":"2018","venue":"Proceedings of the European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Nazeri et al., 2019] Kamyar Nazeri, Eric Ng, Tony Joseph, Faisal Qureshi, and Mehran Ebrahimi. Edgeconnect: Structure guided image inpainting using edge prediction. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 0-0, 2019.","title":"Edgeconnect: Structure guided image inpainting using edge prediction","authors":"Kamyar Nazeri, Eric Ng, Tony Joseph, Faisal Qureshi, Mehran Ebrahimi","year":"2019","venue":"Proceedings of the IEEE International Conference on Computer Vision Workshops","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Newson et al., 2014] Alasdair Newson, Andrés Almansa, Matthieu Fradet, Yann Gousseau, and Patrick Pérez. Video inpainting of complex scenes. SIAM Journal on Imaging Sciences, 7(4):1993-2019, 2014.","title":"Video inpainting of complex scenes","authors":"Alasdair Newson, Andrés Almansa, Matthieu Fradet, Yann Gousseau, Patrick Pérez","year":"2014","venue":"SIAM Journal on Imaging Sciences","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Oh et al., 2019] Seoung Wug Oh, Sungho Lee, Joon-Young Lee, and Seon Joo Kim. Onion-peel networks for deep video completion. In Proceedings of the IEEE Interna- tional Conference on Computer Vision, pages 4403-4412, 2019.","title":"Onion-peel networks for deep video completion","authors":"Oh","year":"2019","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Pathak et al., 2016] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2536-2544, 2016.","title":"Context encoders: Feature learning by inpainting","authors":"Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, Alexei A Efros","year":"2016","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Patwardhan et al., 2005] Kedar A Patwardhan, Guillermo Sapiro, and Marcelo Bertalmio. Video inpainting of oc- cluding and occluded objects. In IEEE International Con- ference on Image Processing, pages 69-72, 2005.","title":"Video inpainting of occluding and occluded objects","authors":"Guillermo Kedar A Patwardhan, Marcelo Sapiro, Bertalmio","year":"2005","venue":"IEEE International Conference on Image Processing","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Patwardhan et al., 2007] Kedar A Patwardhan, Guillermo Sapiro, and Marcelo Bertalmío. Video inpainting under constrained camera motion. IEEE Transactions on Image Processing, 16(2):545-553, 2007.","title":"Video inpainting under constrained camera motion","authors":"Guillermo Kedar A Patwardhan, Marcelo Sapiro, Bertalmío","year":"2007","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Perazzi et al., 2016] Federico Perazzi, Jordi Pont-Tuset, Brian McWilliams, Luc Van Gool, Markus Gross, and Alexander Sorkine-Hornung. A benchmark dataset and evaluation methodology for video object segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 724-732, 2016.","title":"A benchmark dataset and evaluation methodology for video object segmentation","authors":"Federico Perazzi, Jordi Pont-Tuset, Brian Mcwilliams, Luc Van Gool, Markus Gross, Alexander Sorkine-Hornung","year":"2016","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Wang et al., 2019] Chuan Wang, Haibin Huang, Xiaoguang Han, and Jue Wang. Video inpainting by jointly learn- ing temporal structure and spatial details. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 5232-5239, 2019.","title":"Video inpainting by jointly learning temporal structure and spatial details","authors":"Chuan Wang, Haibin Huang, Xiaoguang Han, Jue Wang","year":"2019","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wexler et al., 2007] Yonatan Wexler, Eli Shechtman, and Michal Irani. Space-time completion of video. IEEE Transactions on Pattern Analysis & Machine Intelligence, 29(3):463-476, 2007.","title":"Space-time completion of video","authors":"Yonatan Wexler, Eli Shechtman, Michal Irani","year":"2007","venue":"IEEE Transactions on Pattern Analysis & Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Xu et al., 2018] Ning Xu, Linjie Yang, Yuchen Fan, Dingcheng Yue, Yuchen Liang, Jianchao Yang, and Thomas Huang. Youtube-vos: A large-scale video object segmentation benchmark. arXiv preprint arXiv:1809.03327, 2018.","title":"Youtube-vos: A large-scale video object segmentation benchmark","authors":"Ning Xu, Linjie Yang, Yuchen Fan, Dingcheng Yue, Yuchen Liang, Jianchao Yang, Thomas Huang","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Xu et al., 2019] Rui Xu, Xiaoxiao Li, Bolei Zhou, and Chen Change Loy. Deep flow-guided video inpainting. pages 3723-3732, 2019.","title":"Deep flow-guided video inpainting","authors":"Rui Xu, Xiaoxiao Li, Bolei Zhou, Chen Change Loy","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Yu et al., 2018] Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Generative image inpainting with contextual attention. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition, pages 5505-5514, 2018.","title":"Generative image inpainting with contextual attention","authors":"Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, Thomas S Huang","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhu et al., 2019] Xizhou Zhu, Han Hu, Stephen Lin, and Jifeng Dai. Deformable convnets v2: More deformable, better results. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9308- 9316, 2019.","title":"Deformable convnets v2: More deformable, better results","authors":"Xizhou Zhu, Han Hu, Stephen Lin, Jifeng Dai","year":"2019","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}