{"type":"ScholarlyArticle","title":"Uncertainty-Aware Representation Learning for Action Segmentation","authors":[{"name":"Lei Chen","slug":"lei-chen","given":"Lei","family":"Chen","affiliation":"Beijing National Research Center for Information Science and Technology (BNRist) , and the Department of Automation , Tsinghua University"},{"name":"Muheng Li","slug":"muheng-li","given":"Muheng","family":"Li","affiliation":"Beijing National Research Center for Information Science and Technology (BNRist) , and the Department of Automation , Tsinghua University"},{"name":"Yueqi Duan","slug":"yueqi-duan","given":"Yueqi","family":"Duan","affiliation":"Department of Electronic Engineering , Tsinghua University"},{"name":"Jie Zhou","slug":"jie-zhou","given":"Jie","family":"Zhou","affiliation":"Beijing National Research Center for Information Science and Technology (BNRist) , and the Department of Automation , Tsinghua University"},{"name":"Jiwen Lu","slug":"jiwen-lu","given":"Jiwen","family":"Lu","affiliation":"Beijing National Research Center for Information Science and Technology (BNRist) , and the Department of Automation , Tsinghua University"}],"abstract":"In this paper, we propose an uncertainty-aware representation Learning (UARL) method for action segmentation. Most existing action segmentation methods exploit continuity information of the action period to predict frame-level labels, which ignores the temporal ambiguity of the transition region between two actions. Moreover, similar periods of different actions, e.g., the beginning of some actions, will confuse the network if they are annotated with different labels, which causes spatial ambiguity. To address this, we design the UARL to exploit the transitional expression between two action periods by uncertainty learning. Specially, we model every frame of actions with an active distribution that represents the probabilities of different actions, which captures the uncertainty of the action and exploits the tendency during the action. We evaluate our method on three popular action prediction datasets: Breakfast, Georgia Tech Egocentric Activities (GTEA), and 50Salads. The experimental results demonstrate that our method achieves the performance with state-of-the-art.","doi":"10.24963/ijcai.2022/115","url":"https://ijcai.org/proceedings/2022/115","pdf":"https://ijcai.org/proceedings/2022/0115.pdf","markdown":"https://ijcai.org/proceedings/2022/115.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/115","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":115,"pages":{"first":820,"last":826},"keywords":["Computer Vision: Action and Behaviour Recognition"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986549","length_sec":78,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984692","length_sec":175,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Ahn and Lee, 2021] Hyemin Ahn and Dongheui Lee. Re- fining action segmentation with hierarchical video repre- sentations. In ICCV, pages 16302-16310, 2021.","title":"Refining action segmentation with hierarchical video representations","authors":"Ahn, Lee ;, Hyemin Ahn, Dongheui Lee","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Carreira and Zisserman, 2017] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In CVPR, pages 6299-6308, 2017.","title":"Quo vadis, action recognition? a new model and the kinetics dataset","authors":"Joao Carreira, Andrew Zisserman","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2020] Min-Hung Chen, Baopu Li, Yingze Bao, Ghassan AlRegib, and Zsolt Kira. Action segmentation with joint self-supervised temporal domain adaptation. In CVPR, pages 9454-9463, 2020.","title":"Action segmentation with joint self-supervised temporal domain adaptation","authors":"Chen","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dai et al., 2021] Rui Dai, Srijan Das, and Franc ¸ois Bre- mond. Learning an augmented rgb representation with cross-modal knowledge distillation for action detection. In ICCV, pages 13053-13064, 2021.","title":"Learning an augmented rgb representation with cross-modal knowledge distillation for action detection","authors":"Rui Dai, Srijan Das, Franc, Bremond","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Farha and Gall, 2019] Yazan Abu Farha and Jurgen Gall. Ms-tcn: Multi-stage temporal convolutional network for action segmentation. In CVPR, June 2019.","title":"Ms-tcn: Multi-stage temporal convolutional network for action segmentation","authors":"Farha, Abu Gall ; Yazan, Jurgen Farha, Gall","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fathi et al., 2011] Alireza Fathi, Xiaofeng Ren, and James M Rehg. Learning to recognize objects in egocentric activities. In CVPR, pages 3281-3288, 2011.","title":"Learning to recognize objects in egocentric activities","authors":"Alireza Fathi, Xiaofeng Ren, James M Rehg","year":"2011","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gal and Ghahramani, 2016] Yarin Gal and Zoubin Ghahra- mani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In ICML, pages 1050- 1059, 2016.","title":"Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning","authors":"Ghahramani Gal","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gammulle et al., 2019] Harshala Gammulle, Tharindu Fer- nando, Simon Denman, Sridha Sridharan, and Clinton Fookes. Coupled generative adversarial network for con- tinuous fine-grained action segmentation. In WACV, pages 200-209, 2019.","title":"Coupled generative adversarial network for continuous fine-grained action segmentation","authors":"Gammulle","year":"2019","venue":"WACV","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gao et al., 2021] Shang-Hua Gao, Qi Han, Zhong-Yu Li, Pai Peng, Liang Wang, and Ming-Ming Cheng. Global2local: Efficient structure search for video action segmentation. In CVPR, pages 16805-16814, 2021.","title":"Global2local: Efficient structure search for video action segmentation","authors":"Shang-Hua Gao, Qi Han, Zhong-Yu Li, Pai Peng, Liang Wang, Ming-Ming Cheng","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Guan et al., 2021] Dayan Guan, Jiaxing Huang, Aoran Xiao, and Shijian Lu. Domain adaptive video segmen- tation via temporal consistency regularization. In ICCV, pages 8053-8064, 2021.","title":"Domain adaptive video segmentation via temporal consistency regularization","authors":"Dayan Guan, Jiaxing Huang, Aoran Xiao, Shijian Lu","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hilde et al., 2014] Kuehne Hilde, Bilgin Arslan Ali, and Serre Thomas. The language of actions: Recovering the syntax and semantics of goal-directed human activities. In CVPR, pages 780-787, 2014.","title":"The language of actions: Recovering the syntax and semantics of goal-directed human activities","authors":"Bilgin Arslan Kuehne Hilde, Serre Ali, Thomas","year":"2014","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Ishikawa et al., 2021] Yuchi Ishikawa, Seito Kasai, Yoshimitsu Aoki, and Hirokatsu Kataoka. Alleviating over-segmentation errors by detecting action boundaries. In WACV, pages 2322-2331, 2021.","title":"Alleviating over-segmentation errors by detecting action boundaries","authors":"Ishikawa","year":"2021","venue":"WACV","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kendall and Gal, 2017] Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? In NeurIPS, 2017.","title":"What uncertainties do we need in bayesian deep learning for computer vision?","authors":"Alex Kendall, Yarin Gal","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kendall et al., 2015] Alex Kendall, Vijay Badrinarayanan, and Roberto Cipolla. Bayesian segnet: Model uncertainty in deep convolutional encoder-decoder architectures for scene understanding. In BMVC, 2015.","title":"Bayesian segnet: Model uncertainty in deep convolutional encoder-decoder architectures for scene understanding","authors":"Alex Kendall, Vijay Badrinarayanan, Roberto Cipolla","year":"2015","venue":"BMVC","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kingma and Welling, 2014] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014.","title":"Auto-encoding variational bayes","authors":"P Diederik, Max Kingma, Welling","year":"2014","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lea et al., 2016] Colin Lea, Austin Reiter, René Vidal, and Gregory D Hager. Segmental spatiotemporal cnns for fine-grained action segmentation. In ECCV, pages 36-52, 2016.","title":"Segmental spatiotemporal cnns for fine-grained action segmentation","authors":"Colin Lea, Austin Reiter, René Vidal, Gregory D Hager","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lea et al., 2017a] Colin Lea, Michael D. Flynn, Rene Vidal, Austin Reiter, and Gregory D. Hager. Temporal convolu- tional networks for action segmentation and detection. In CVPR, 2017.","title":"Temporal convolutional networks for action segmentation and detection","authors":"Lea","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lea et al., 2017b] Colin Lea, Michael D Flynn, Rene Vidal, Austin Reiter, and Gregory D Hager. Temporal convolu- tional networks for action segmentation and detection. In CVPR, pages 156-165, 2017.","title":"Temporal convolutional networks for action segmentation and detection","authors":"Lea","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lei and Todorovic, 2018] Peng Lei and Sinisa Todorovic. Temporal deformable residual networks for action seg- mentation in videos. In CVPR, pages 6742-6751, 2018.","title":"Temporal deformable residual networks for action segmentation in videos","authors":"Todorovic ; ] Lei, Lei Peng, Sinisa Todorovic","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Li et al., 2020] Shi-Jie Li, Yazan AbuFarha, Yun Liu, Ming- Ming Cheng, and Juergen Gall. Ms-tcn++: Multi-stage temporal convolutional network for action segmentation. TPAMI, pages 1-1, 2020.","title":"Ms-tcn++: Multi-stage temporal convolutional network for action segmentation","authors":"Shi-Jie Li, Yazan Abufarha, Yun Liu, Ming-Ming Cheng, Juergen Gall","year":"2020","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Li et al., 2021] Shuang Li, Yilun Du, Antonio Torralba, Josef Sivic, and Bryan Russell. Weakly supervised human- object interaction detection in video via contrastive spa- tiotemporal regions. In ICCV, pages 1845-1855, 2021.","title":"Weakly supervised humanobject interaction detection in video via contrastive spatiotemporal regions","authors":"Shuang Li, Yilun Du, Antonio Torralba, Josef Sivic, Bryan Russell","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[MacKay, 1992] David JC MacKay. A practical bayesian framework for backpropagation networks. NC, 4(3):448- 472, 1992.","title":"A practical bayesian framework for backpropagation networks","authors":"J C David, Mackay","year":"1992","venue":"NC","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Menapace et al., 2021] Willi Menapace, Stephane Lath- uiliere, Sergey Tulyakov, Aliaksandr Siarohin, and Elisa Ricci. Playable video generation. In CVPR, pages 10061- 10070, 2021.","title":"Playable video generation","authors":"Willi Menapace, Stephane Lathuiliere, Sergey Tulyakov, Aliaksandr Siarohin, Elisa Ricci","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Sridhar et al., 2021] Deepak Sridhar, Niamul Quader, Srikanth Muralidharan, Yaoxin Li, Peng Dai, and Juwei Lu. Class semantics-based attention for action detection. In ICCV, pages 13739-13748, 2021.","title":"Class semantics-based attention for action detection","authors":"Deepak Sridhar, Niamul Quader, Srikanth Muralidharan, Yaoxin Li, Peng Dai, Juwei Lu","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Stein and McKenna, 2013] Sebastian Stein and Stephen J McKenna. Combining embedded accelerometers with computer vision for recognizing food preparation activi- ties. In ACM IJCPUC, pages 729-738, 2013.","title":"Combining embedded accelerometers with computer vision for recognizing food preparation activities","authors":"Mckenna Stein, Sebastian Stein, Stephen J Mckenna","year":"2013","venue":"ACM IJCPUC","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Sultani et al., 2018] Waqas Sultani, Chen Chen, and Mubarak Shah. Real-world anomaly detection in surveillance videos. In CVPR, pages 2942-2950, 2018.","title":"Real-world anomaly detection in surveillance videos","authors":"Waqas Sultani, Chen Chen, Mubarak Shah","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NuerIPS, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NuerIPS","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wang et al., 2020] Zhenzhi Wang, Ziteng Gao, Limin Wang, Zhifeng Li, and Gangshan Wu. Boundary-aware cascade networks for temporal action segmentation. In ECCV, pages 34-51, 2020.","title":"Boundary-aware cascade networks for temporal action segmentation","authors":"Wang","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Yi et al., 2021] Fangqiu Yi, Hongyu Wen, and Tingting Jiang. Asformer: Transformer for action segmentation. In BMVC, pages 1-15, 2021.","title":"Asformer: Transformer for action segmentation","authors":"Yi","year":"2021","venue":"BMVC","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}