{"type":"ScholarlyArticle","title":"Region-Aware Temporal Inconsistency Learning for DeepFake Video Detection","authors":[{"name":"Zhihao Gu","slug":"zhihao-gu","given":"Zhihao","family":"Gu","affiliation":"School of Electronic and Electrical Engineering , Shanghai Jiao Tong University , Shanghai , China; Tencent Youtu Lab , Shanghai , China"},{"name":"Taiping Yao","slug":"taiping-yao","given":"Taiping","family":"Yao","affiliation":"Tencent Youtu Lab , Shanghai , China"},{"name":"Yang Chen","slug":"yang-chen","given":"Yang","family":"Chen","affiliation":"Tencent Youtu Lab , Shanghai , China"},{"name":"Ran Yi","slug":"ran-yi","given":"Ran","family":"Yi","affiliation":"School of Electronic and Electrical Engineering , Shanghai Jiao Tong University , Shanghai , China"},{"name":"Shouhong Ding","slug":"shouhong-ding","given":"Shouhong","family":"Ding","affiliation":"Tencent Youtu Lab , Shanghai , China"},{"name":"Lizhuang Ma","slug":"lizhuang-ma","given":"Lizhuang","family":"Ma","affiliation":"School of Electronic and Electrical Engineering , Shanghai Jiao Tong University , Shanghai , China; MoE Key Lab of Artificial Intelligence , Shanghai Jiao Tong University , Shanghai , China"}],"abstract":"The rapid development of face forgery techniques has drawn growing attention due to security concerns.  Existing deepfake video detection methods always attempt to capture the discriminative features by directly exploiting static temporal convolution to mine temporal inconsistency, without explicit exploration on the diverse temporal dynamics of different forged regions. To effectively and comprehensively capture the various inconsistency, in this paper, we propose a novel Region-Aware Temporal Filter (RATF) module which automatically generates corresponding temporal filters for different spatial regions. Specifically, we decouple the dynamic temporal kernel into a set of region-agnostic basic filters and region-sensitive aggregation weights. And different weights guide the corresponding regions to adaptively learn temporal inconsistency, which greatly enhances the overall representational ability. Moreover, to cover the long-term temporal dynamics, we divide the video into multiple snippets and propose a Cross-Snippet Attention (CSA) to promote the cross-snippet information interaction. Extensive experiments and visualizations on several benchmarks demonstrate the effectiveness of our method against state-of-the-art competitors.","doi":"10.24963/ijcai.2022/129","url":"https://ijcai.org/proceedings/2022/129","pdf":"https://ijcai.org/proceedings/2022/0129.pdf","markdown":"https://ijcai.org/proceedings/2022/129.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/129","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":129,"pages":{"first":920,"last":926},"keywords":["Computer Vision: Biometrics, Face, Gesture and Pose Recognition"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986508","length_sec":84,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984651","length_sec":279,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Carreira and Zisserman, 2017] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In CVPR, 2017.","title":"Quo vadis, action recognition? a new model and the kinetics dataset","authors":"Joao Carreira, Andrew Zisserman","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2021] Shen Chen, Taiping Yao, Yang Chen, Shouhong Ding, Jilin Li, and Rongrong Ji. Local relation learning for face forgery detection. In AAAI, 2021.","title":"Local relation learning for face forgery detection","authors":"Taiping Shen Chen, Yang Yao, Shouhong Chen, Jilin Ding, Rongrong Li, Ji","year":"2021","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Dolhansky et al., 2019] Brian Dolhansky, Russ Howes, Ben Pflaum, Nicole Baram, and Cristian Canton Ferrer. The deepfake detection challenge (dfdc) preview dataset. In arXiv, abs/1910.08854, 2019.","title":"The deepfake detection challenge (dfdc) preview dataset","authors":"Brian Dolhansky, Russ Howes, Ben Pflaum, Nicole Baram, Cristian Canton Ferrer","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Gu et al., 2021a] Qiqi Gu, Shen Chen, Taiping Yao, Yang Chen, Shouhong Ding, and Ran Yi. Exploiting fine- grained face forgery clues via progressive enhancement learning. In AAAI, 2021.","title":"Exploiting finegrained face forgery clues via progressive enhancement learning","authors":"Gu","year":"2021","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Gu et al., 2021b] Zhihao Gu, Yang Chen, Taiping Yao, Shouhong Ding, Jilin Li, Feiyue Huang, and Lizhuang Ma. Spatiotemporal inconsistency learning for deepfake video detection. In ACM MM, 2021.","title":"Spatiotemporal inconsistency learning for deepfake video detection","authors":"Gu","year":"2021","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hu et al., 2021] Ziheng Hu, Hongtao Xie, Yuxin Wang, Ji- ahong Li, Zhongyuan Wang, and Yongdong Zhang. Dy- namic inconsistency-aware deepfake video detection. In IJCAI, 2021.","title":"Dynamic inconsistency-aware deepfake video detection","authors":"Ziheng Hu, Hongtao Xie, Yuxin Wang, Jiahong Li, Zhongyuan Wang, Yongdong Zhang","year":"2021","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2021/102"},{"position":7,"raw":"[Jia et al., 2016] Xu Jia, Bert De Brabandere, Tinne Tuyte- laars, and Luc V Gool. Dynamic filter networks. In NIPS, 2016.","title":"Dynamic filter networks","authors":"Xu Jia, Bert De Brabandere, Tinne Tuytelaars, Luc V Gool","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Li et al., 2020a] Lingzhi Li, Jianmin Bao, Ting Zhang, Hao Yang, Dong Chen, Fang Wen, and Baining Guo. Face x-ray for more general face forgery detection. In CVPR, 2020.","title":"Face x-ray for more general face forgery detection","authors":"Li","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Li et al., 2020b] Xiaodan Li, Yining Lang, Yuefeng Chen, Xiaofeng Mao, Yuan He, Shuhui Wang, Hui Xue, and Quan Lu. Sharp multiple instance learning for deepfake video detection. In ACM MM, 2020.","title":"Sharp multiple instance learning for deepfake video detection","authors":"Li","year":"2020","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li et al., 2020c] Yuezun Li, Xin Yang, Pu Sun, Honggang Qi, and Siwei Lyu. Celeb-df: A large-scale challenging dataset for deepfake forensics. In CVPR, 2020.","title":"Celeb-df: A large-scale challenging dataset for deepfake forensics","authors":"Li","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Liu et al., 2020] Zhaoyang Liu, Donghao Luo, Yabiao Wang, Limin Wang, Ying Tai, Chengjie Wang, Jilin Li, Feiyue Huang, and Tong Lu. Teinet: Towards an efficient architecture for video recognition. In AAAI, 2020.","title":"Teinet: Towards an efficient architecture for video recognition","authors":"Zhaoyang Liu, Donghao Luo, Yabiao Wang, Limin Wang, Ying Tai, Chengjie Wang, Jilin Li, Feiyue Huang, Tong Lu","year":"2020","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Liu et al., 2021] Zhaoyang Liu, Limin Wang, Wayne Wu, Chen Qian, and Tong Lu. Tam: Temporal adaptive module for video recognition. In ICCV, 2021.","title":"Tam: Temporal adaptive module for video recognition","authors":"Liu","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Masi et al., 2020] Iacopo Masi, Aditya Killekar, Roys- ton Marian Mascarenhas, Shenoy Pratik Gurudatt, and Wael AbdAlmageed. Two-branch recurrent network for isolating deepfakes in videos. In ECCV, 2020.","title":"Two-branch recurrent network for isolating deepfakes in videos","authors":"Iacopo Masi, Aditya Killekar, Marian Royston, Shenoy Mascarenhas, Wael Pratik Gurudatt, Abdalmageed","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Qi et al., 2020] Hua Qi, Qing Guo, Felix Juefei-Xu, Xiaofei Xie, Lei Ma, Wei Feng, Yang Liu, and Jianjun Zhao. Deep- rhythm: Exposing deepfakes with attentional visual heart- beat rhythms. In ACM MM, 2020.","title":"Deeprhythm: Exposing deepfakes with attentional visual heartbeat rhythms","authors":"Hua Qi, Qing Guo, Felix Juefei-Xu, Xiaofei Xie, Lei Ma, Wei Feng, Yang Liu, Jianjun Zhao","year":"2020","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Rossler et al., 2019] Andreas Rossler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, and Matthias Nießner. Faceforensics++: Learning to detect manipulated facial images. In ICCV, 2019.","title":"Faceforensics++: Learning to detect manipulated facial images","authors":"Andreas Rossler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, Matthias Nießner","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Selvaraju et al., 2017] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In ICCV, 2017.","title":"Grad-cam: Visual explanations from deep networks via gradient-based localization","authors":"Michael Ramprasaath R Selvaraju, Abhishek Cogswell, Ramakrishna Das, Devi Vedantam, Dhruv Parikh, Batra","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Sohrawardi et al., 2019] Saniat Javid Sohrawardi, Akash Chintha, Bao Thai, Sovantharith Seng, Andrea Hickerson, Raymond Ptucha, and Matthew Wright. Poster: Towards robust open-world detection of deepfakes. In CCS, 2019.","title":"Poster: Towards robust open-world detection of deepfakes","authors":"Sohrawardi","year":"2019","venue":"CCS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Sun et al., 2022] Ke Sun, Taiping Yao, Shen Chen, Shouhong Ding, Rongrong Ji, et al. Dual contrastive learn- ing for general face forgery detection. In AAAI, 2022.","title":"Dual contrastive learning for general face forgery detection","authors":"Ke Sun, Taiping Yao, Shen Chen, Shouhong Ding, Rongrong Ji","year":"2022","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wang et al., 2018] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In CVPR, 2018.","title":"Non-local neural networks","authors":"Xiaolong Wang, Ross Girshick, Abhinav Gupta, Kaiming He","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang et al., 2020a] Gengxing Wang, Jiahuan Zhou, and Ying Wu. Exposing deep-faked videos by anomalous co- motion pattern detection. In arXiv, abs/2008.04848, 2020.","title":"Exposing deep-faked videos by anomalous comotion pattern detection","authors":"Wang","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2020b] Xinyao Wang, Taiping Yao, Shouhong Ding, and Lizhuang Ma. Face manipulation detection via auxiliary supervision. In ICONIP, 2020.","title":"Face manipulation detection via auxiliary supervision","authors":"Wang","year":"2020","venue":"ICONIP","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wu et al., 2021] Wenhao Wu, Yuxiang Zhao, Yanwu Xu, Xiao Tan, Dongliang He, Zhikang Zou, Jin Ye, Yingying Li, Mingde Yao, Zichao Dong, et al. Dsanet: Dynamic segment aggregation network for video-level representa- tion learning. In ACM MM, 2021.","title":"Dsanet: Dynamic segment aggregation network for video-level representation learning","authors":"Wenhao Wu, Yuxiang Zhao, Yanwu Xu, Xiao Tan, Dongliang He, Zhikang Zou, Jin Ye, Yingying Li, Mingde Yao, Zichao Dong","year":"2021","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhang et al., 2020] Shiwen Zhang, Sheng Guo, Weilin Huang, Matthew R Scott, and Limin Wang. V4d: 4d con- volutional neural networks for video-level representation learning. In arXiv, abs/2002.07442, 2020.","title":"V4d: 4d convolutional neural networks for video-level representation learning","authors":"Shiwen Zhang, Sheng Guo, Weilin Huang, Matthew R Scott, Limin Wang","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhang et al., 2021] Daichi Zhang, Chenyu Li, Fanzhao Lin, Dan Zeng, and Shiming Ge. Detecting deepfake videos with temporal dropout 3dcnn. In IJCAI, 2021.","title":"Detecting deepfake videos with temporal dropout 3dcnn","authors":"Daichi Zhang, Chenyu Li, Fanzhao Lin, Dan Zeng, Shiming Ge","year":"2021","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2021/178"},{"position":25,"raw":"[Zhou et al., 2021] Jingkai Zhou, Varun Jampani, Zhixiong Pi, Qiong Liu, and Ming-Hsuan Yang. Decoupled dynamic filter networks. In CVPR, 2021.","title":"Decoupled dynamic filter networks","authors":"Zhou","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zi et al., 2020] Bojia Zi, Minghao Chang, Jingjing Chen, Xingjun Ma, and Yu-Gang Jiang. Wilddeepfake: A chal- lenging real-world dataset for deepfake detection. In ACM MM, 2020.","title":"Wilddeepfake: A challenging real-world dataset for deepfake detection","authors":"Zi","year":"2020","venue":"ACM MM","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}