{"type":"ScholarlyArticle","title":"PACE: Predictive and Contrastive Embedding for Unsupervised Action Segmentation","authors":[{"name":"Jiahao Wang","slug":"jiahao-wang","given":"Jiahao","family":"Wang","affiliation":"State Key Laboratory of Virtual Reality Technology and System , Beihang University"},{"name":"Jie Qin","slug":"jie-qin","given":"Jie","family":"Qin","affiliation":"College of Computer Science and Technology , Nanjing University of Aeronautics and Astronautics"},{"name":"Yunhong Wang","slug":"yunhong-wang","given":"Yunhong","family":"Wang","affiliation":"State Key Laboratory of Virtual Reality Technology and System , Beihang University"},{"name":"Annan Li","slug":"annan-li","given":"Annan","family":"Li","affiliation":"State Key Laboratory of Virtual Reality Technology and System , Beihang University"}],"abstract":"Action segmentation, inferring temporal positions of human actions in an untrimmed video, is an important prerequisite for various video understanding tasks. Recently, unsupervised action segmentation (UAS) has emerged as a more challenging task due to the unavailability of frame-level annotations. Existing clustering- or prediction-based UAS approaches suffer from either over-segmentation or overfitting, leading to unsatisfactory results. To address those problems,we propose Predictive And Contrastive Embedding (PACE), a unified UAS framework leveraging both predictability and similarity information for more accurate action segmentation. On the basis of an auto-regressive transformer encoder, predictive embeddings are learned by exploiting the predictability of video context, while contrastive embeddings are generated by leveraging the similarity of adjacent short video clips. Extensive experiments on three challenging benchmarks demonstrate the superiority of our method, with up to 26.9% improvements in F1-score over the state of the art.","doi":"10.24963/ijcai.2022/198","url":"https://ijcai.org/proceedings/2022/198","pdf":"https://ijcai.org/proceedings/2022/0198.pdf","markdown":"https://ijcai.org/proceedings/2022/198.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/198","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":198,"pages":{"first":1423,"last":1429},"keywords":["Computer Vision: Action and Behaviour Recognition","Computer Vision: Transfer, low-shot, semi- and un- supervised learning","Computer Vision: Video analysis and understanding"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986461","length_sec":90,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984604","length_sec":321,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Aakur and Sarkar, 2019] Sathyanarayanan N Aakur and Sudeep Sarkar. A perceptual prediction framework for self supervised event segmentation. In Proc. CVPR, pages 1197-1206, 2019.","title":"Sathyanarayanan N Aakur and Sudeep Sarkar. A perceptual prediction framework for self supervised event segmentation","authors":"Sarkar Aakur","year":"2019","venue":"Proc. CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Alayrac et al., 2016] Jean-Baptiste Alayrac, Piotr Bo- janowski, Nishant Agrawal, Josef Sivic, Ivan Laptev, and Simon Lacoste-Julien. Unsupervised learning from narrated instruction videos. In Proc. CVPR, pages 4575-4583, 2016.","title":"Unsupervised learning from narrated instruction videos","authors":"Jean-Baptiste Alayrac, Piotr Bojanowski, Nishant Agrawal, Josef Sivic, Ivan Laptev, Simon Lacoste-Julien","year":"2016","venue":"Proc. CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bhatnagar et al., 2017] Bharat Lal Bhatnagar, Suriya Singh, Chetan Arora, and CV Jawahar. Unsupervised learning of deep feature representation for clustering egocentric ac- tions. In Proc. IJCAI, pages 1447-1453, 2017.","title":"Unsupervised learning of deep feature representation for clustering egocentric actions","authors":"Bharat Lal Bhatnagar, Suriya Singh, Chetan Arora, Jawahar","year":"2017","venue":"Proc. IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2017/200"},{"position":4,"raw":"[Carreira and Zisserman, 2017] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In Proc. CVPR, pages 6299-6308, 2017.","title":"Quo vadis, action recognition? a new model and the kinetics dataset","authors":"Joao Carreira, Andrew Zisserman","year":"2017","venue":"Proc. CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Farha and Gall, 2019] Yazan Abu Farha and Jurgen Gall. Ms-tcn: Multi-stage temporal convolutional network for action segmentation. In Proc. CVPR, pages 3575-3584, 2019.","title":"Ms-tcn: Multi-stage temporal convolutional network for action segmentation","authors":"Farha, Abu Gall ; Yazan, Jurgen Farha, Gall","year":"2019","venue":"Proc. CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Kingma and Ba, 2015] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proc. ICLR, 2015.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2015","venue":"Proc. ICLR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Kuehne et al., 2014] Hilde Kuehne, Ali Arslan, and Thomas Serre. The language of actions: Recovering the syntax and semantics of goal-directed human activities. In Proc. CVPR, pages 780-787, 2014.","title":"The language of actions: Recovering the syntax and semantics of goal-directed human activities","authors":"Hilde Kuehne, Ali Arslan, Thomas Serre","year":"2014","venue":"Proc. CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kukleva et al., 2019] Anna Kukleva, Hilde Kuehne, Fadime Sener, and Jurgen Gall. Unsupervised learning of action classes with continuous temporal embedding. In Proc. CVPR, pages 12066-12074, 2019.","title":"Unsupervised learning of action classes with continuous temporal embedding","authors":"Anna Kukleva, Hilde Kuehne, Fadime Sener, Jurgen Gall","year":"2019","venue":"Proc. CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Lea et al., 2017] Colin Lea, Michael D Flynn, Rene Vidal, Austin Reiter, and Gregory D Hager. Temporal convolu- tional networks for action segmentation and detection. In Proc. CVPR, pages 156-165, 2017.","title":"Temporal convolutional networks for action segmentation and detection","authors":"Colin Lea, Rene Michael D Flynn, Austin Vidal, Gregory D Reiter, Hager","year":"2017","venue":"Proc. CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li and Todorovic, 2020] Jun Li and Sinisa Todorovic. Set- constrained viterbi for set-supervised action segmentation. In Proc. CVPR, pages 10820-10829, 2020.","title":"Setconstrained viterbi for set-supervised action segmentation","authors":"Li, Todorovic, Jun Li, Sinisa Todorovic","year":"2020","venue":"Proc. CVPR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li and Todorovic, 2021] Jun Li and Sinisa Todorovic. Ac- tion shuffle alternating learning for unsupervised action segmentation. In Proc. CVPR, pages 12628-12636, 2021.","title":"Action shuffle alternating learning for unsupervised action segmentation","authors":"Li, Todorovic, Jun Li, Sinisa Todorovic","year":"2021","venue":"Proc. CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2021] Yixuan Li, Lei Chen, Runyu He, Zhenzhi Wang, Gangshan Wu, and Limin Wang. Multisports: A multi-person video dataset of spatio-temporally localized sports actions. In Proc. ICCV, pages 13536-13545, 2021.","title":"Multisports: A multi-person video dataset of spatio-temporally localized sports actions","authors":"Yixuan Li, Lei Chen, Runyu He, Zhenzhi Wang, Gangshan Wu, Limin Wang","year":"2021","venue":"Proc. ICCV","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mnih and Kavukcuoglu, 2013] Andriy Mnih and Koray Kavukcuoglu. Learning word embeddings efficiently with noise-contrastive estimation. In Proc. NeurIPS, pages 2265-2273, 2013.","title":"Learning word embeddings efficiently with noise-contrastive estimation","authors":"Andriy Mnih, Koray Kavukcuoglu","year":"2013","venue":"Proc. NeurIPS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Richard et al., 2018] Alexander Richard, Hilde Kuehne, Ahsan Iqbal, and Juergen Gall. Neuralnetwork-viterbi: A framework for weakly supervised video learning. In Proc. CVPR, pages 7386-7395, 2018.","title":"Neuralnetwork-viterbi: A framework for weakly supervised video learning","authors":"Alexander Richard, Hilde Kuehne, Ahsan Iqbal, Juergen Gall","year":"2018","venue":"Proc. CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Sarfraz et al., 2021] Saquib Sarfraz, Naila Murray, Vivek Sharma, Ali Diba, Luc Van Gool, and Rainer Stiefelhagen. Temporally-weighted hierarchical clustering for unsuper- vised action segmentation. In Proc. CVPR, pages 11225- 11234, 2021.","title":"Temporally-weighted hierarchical clustering for unsupervised action segmentation","authors":"Saquib Sarfraz, Naila Murray, Vivek Sharma, Ali Diba, Luc Van Gool, Rainer Stiefelhagen","year":"2021","venue":"Proc. CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Sener and Yao, 2018] Fadime Sener and Angela Yao. Unsu- pervised learning and segmentation of complex activities from video. In Proc. CVPR, pages 8368-8376, 2018.","title":"Unsupervised learning and segmentation of complex activities from video","authors":"Yao Sener, Fadime Sener, Angela Yao","year":"2018","venue":"Proc. CVPR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Singh et al., 2016] Bharat Singh, Tim K Marks, Michael Jones, Oncel Tuzel, and Ming Shao. A multi-stream bi- directional recurrent neural network for fine-grained ac- tion detection. In Proc. CVPR, pages 1961-1970, 2016.","title":"A multi-stream bidirectional recurrent neural network for fine-grained action detection","authors":"Bharat Singh, Tim K Marks, Michael Jones, Oncel Tuzel, Ming Shao","year":"2016","venue":"Proc. CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Stein and McKenna, 2013] Sebastian Stein and Stephen J McKenna. Combining embedded accelerometers with computer vision for recognizing food preparation activi- ties. In Proc. UbiComp, pages 729-738, 2013.","title":"Combining embedded accelerometers with computer vision for recognizing food preparation activities","authors":"Mckenna Stein, Sebastian Stein, Stephen J Mckenna","year":"2013","venue":"Proc. UbiComp","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Van der Maaten and Hinton, 2008] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9(11), 2008.","title":"Visualizing data using t-sne","authors":"Van Der Maaten, Laurens Hinton, Geoffrey Van Der Maaten, Hinton","year":"2008","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Virtanen et al., 2020] Pauli Virtanen, Ralf Gommers, Travis E Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, et al. Scipy 1.0: Fundamental algorithms for scientific computing in python. Nature Methods, 17(3):261-272, 2020.","title":"Scipy 1.0: Fundamental algorithms for scientific computing in python","authors":"Pauli Virtanen, Ralf Gommers, Travis E Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright","year":"2020","venue":"Nature Methods","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang and Schmid, 2013] Heng Wang and Cordelia Schmid. Action recognition with improved trajectories. In Proc. ICCV, pages 3551-3558, 2013.","title":"Action recognition with improved trajectories","authors":"Wang, Schmid ; ] Heng, Wang, Cordelia Schmid","year":"2013","venue":"Proc. ICCV","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang et al., 2019] Jiahao Wang, Zhengyin Du, Annan Li, and Yunhong Wang. Atrous temporal convolutional net- work for video action segmentation. In Proc. ICIP, pages 1585-1589. IEEE, 2019.","title":"Atrous temporal convolutional network for video action segmentation","authors":"Jiahao Wang, Zhengyin Du, Annan Li, Yunhong Wang","year":"2019","venue":"Proc. ICIP","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2020] Zhenzhi Wang, Ziteng Gao, Limin Wang, Zhifeng Li, and Gangshan Wu. Boundary-aware cascade networks for temporal action segmentation. In Proc. ECCV, pages 34-51. Springer, 2020.","title":"Boundary-aware cascade networks for temporal action segmentation","authors":"Wang","year":"2020","venue":"Proc. ECCV","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang et al., 2021] Jiahao Wang, Yunhong Wang, Sheng Liu, and Annan Li. Few-shot fine-grained action recog- nition via bidirectional attention and contrastive meta- learning. In Proc. ACM MM, pages 582-591, 2021.","title":"Few-shot fine-grained action recognition via bidirectional attention and contrastive metalearning","authors":"Jiahao Wang, Yunhong Wang, Sheng Liu, Annan Li","year":"2021","venue":"Proc. ACM MM","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wang et al., 2022] Zhe Wang, Hao Chen, Xinyu Li, Chun- hui Liu, Yuanjun Xiong, Joseph Tighe, and Charless C Fowlkes. Sscap: Self-supervised co-occurrence action parsing for unsupervised temporal action segmentation. In Proc. WACV, volume 2022, 2022.","title":"Sscap: Self-supervised co-occurrence action parsing for unsupervised temporal action segmentation","authors":"Zhe Wang, Hao Chen, Xinyu Li, Chunhui Liu, Yuanjun Xiong, Joseph Tighe, Charless C Fowlkes","year":"2022","venue":"Proc. WACV","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zacks et al., 2001] Jeffrey M Zacks, Barbara Tversky, and Gowri Iyer. Perceiving, remembering, and communicating structure in events. Journal of Experimental Psychology: General, 130(1):29, 2001.","title":"Perceiving, remembering, and communicating structure in events","authors":"Barbara Jeffrey M Zacks, Gowri Tversky, Iyer","year":"2001","venue":"Journal of Experimental Psychology: General","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}