What If We Could Not See? Counterfactual Analysis for Egocentric Action Anticipation

Zhang, Tianyu; Min, Weiqing; Yang, Jiahao; Liu, Tao; Jiang, Shuqiang; Rui, Yong

doi:10.24963/ijcai.2021/182

What If We Could Not See? Counterfactual Analysis for Egocentric Action Anticipation

Tianyu Zhang, Weiqing Min, Jiahao Yang, Tao Liu, Shuqiang Jiang, Yong Rui

Download BibTeX Cite

Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence. Main Track. Pages 1316-1322.

https://doi.org/10.24963/ijcai.2021/182

Keywords

Computer Vision: Video: Events, Activities and Surveillance

Abstract

Egocentric action anticipation aims at predicting the near future based on past observation in first-person vision. While future actions may be wrongly predicted due to the dataset bias, we present a counterfactual analysis framework for egocentric action anticipation (CA-EAA) to enhance the capacity. In the factual case, we can predict the upcoming action based on visual features and semantic labels from past observation. Imagining one counterfactual situation where no visual representation had been observed, we would obtain a counterfactual predicted action only using past semantic labels. In this way, we can reduce the side-effect caused by semantic labels via a comparison between factual and counterfactual outcomes, which moves a step towards unbiased prediction for egocentric action anticipation. We conduct experiments on two large-scale egocentric video datasets. Qualitative and quantitative results validate the effectiveness of our proposed CA-EAA.

Tianyu Zhang, Weiqing Min, Jiahao Yang, Tao Liu, Shuqiang Jiang, and Yong Rui. 2021. What If We Could Not See? Counterfactual Analysis for Egocentric Action Anticipation. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21. IJCAI, 1316–1322. https://doi.org/10.24963/ijcai.2021/182

@inproceedings{ijcai2021p182,
  title     = {What If We Could Not See? Counterfactual Analysis for Egocentric Action Anticipation},
  author    = {Zhang, Tianyu and Min, Weiqing and Yang, Jiahao and Liu, Tao and Jiang, Shuqiang and Rui, Yong},
  booktitle = {Proceedings of the Thirtieth International Joint Conference on
               Artificial Intelligence, {IJCAI-21}},
  publisher = {International Joint Conferences on Artificial Intelligence Organization},
  editor    = {Prof. Zhi-Hua Zhou},
  pages     = {1316--1322},
  year      = {2021},
  month     = {8},
  note      = {Main Track},
  doi       = {10.24963/ijcai.2021/182},
  url       = {https://doi.org/10.24963/ijcai.2021/182},
}

TY  - CPAPER
TI  - What If We Could Not See? Counterfactual Analysis for Egocentric Action Anticipation
AU  - Zhang, Tianyu
AU  - Min, Weiqing
AU  - Yang, Jiahao
AU  - Liu, Tao
AU  - Jiang, Shuqiang
AU  - Rui, Yong
T2  - Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21
PB  - International Joint Conferences on Artificial Intelligence Organization
PY  - 2021
SP  - 1316
EP  - 1322
DO  - 10.24963/ijcai.2021/182
UR  - https://doi.org/10.24963/ijcai.2021/182
ER  -