{"type":"ScholarlyArticle","title":"Timestamp-Supervised Action Segmentation from the Perspective of Clustering","authors":[{"name":"Dazhao Du","slug":"dazhao-du","given":"Dazhao","family":"Du","affiliation":"Institute of Software , Chinese Academy of Science; University of Chinese Academy of Sciences"},{"name":"Enhan Li","slug":"enhan-li","given":"Enhan","family":"Li","affiliation":"University of Chinese Academy of Sciences; Institute of Computing Technology , Chinese Academy of Sciences"},{"name":"Lingyu Si","slug":"lingyu-si","given":"Lingyu","family":"Si","affiliation":"Institute of Software , Chinese Academy of Science; University of Chinese Academy of Sciences"},{"name":"Fanjiang Xu","slug":"fanjiang-xu","given":"Fanjiang","family":"Xu","affiliation":"Institute of Software , Chinese Academy of Science"},{"name":"Fuchun Sun","slug":"fuchun-sun","given":"Fuchun","family":"Sun","affiliation":"Institute of Software , Chinese Academy of Science; Department of Computer Science and Technology , Tsinghua University"}],"abstract":"Video action segmentation under timestamp supervision has recently received much attention due to lower annotation costs. Most existing methods generate pseudo-labels for all frames in each video to train the segmentation model. However, these methods suffer from incorrect pseudo-labels, especially for the semantically unclear frames in the transition region between two consecutive actions, which we call ambiguous intervals. To address this issue, we propose a novel framework from the perspective of clustering, which includes the following two parts. First, pseudo-label ensembling generates incomplete but high-quality pseudo-label sequences, where the frames in ambiguous intervals have no pseudo-labels. Second, iterative clustering iteratively propagates the pseudo-labels to the ambiguous intervals by clustering, and thus updates the pseudo-label sequences to train the model. We further introduce a clustering loss, which encourages the features of frames within the same action segment more compact. Extensive experiments show the effectiveness of our method.","doi":"10.24963/ijcai.2023/77","url":"https://ijcai.org/proceedings/2023/77","pdf":"https://ijcai.org/proceedings/2023/0077.pdf","markdown":"https://ijcai.org/proceedings/2023/77.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/77","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Computer Vision","track":"Main Track","number":77,"pages":{"first":690,"last":698},"keywords":["Computer Vision: CV: Machine learning for vision","Computer Vision: CV: Video analysis and understanding","Computer Vision: CV: Applications"],"videos":[],"references":[{"position":1,"raw":"[Behrmann et al., 2022] Nadine Behrmann, S. Alireza Golestaneh, Zico Kolter, Juergen Gall, and Mehdi Noroozi. Unified fully and timestamp supervised tem- poral action segmentation via sequence to sequence translation. In ECCV, 2022.","title":"Unified fully and timestamp supervised temporal action segmentation via sequence to sequence translation","authors":"Nadine Behrmann, S Alireza Golestaneh, Zico Kolter, Juergen Gall, Mehdi Noroozi","year":"2022","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Carreira and Zisserman, 2017] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6299-6308, 2017.","title":"Quo vadis, action recognition? a new model and the kinetics dataset","authors":"Joao Carreira, Andrew Zisserman","year":"2017","venue":"proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chang et al., 2019] Chien-Yi Chang, De-An Huang, Yanan Sui, Li Fei-Fei, and Juan Carlos Niebles. D3tw: Discrimi- native differentiable dynamic time warping for weakly su- pervised action alignment and segmentation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3546-3555, 2019.","title":"D3tw: Discriminative differentiable dynamic time warping for weakly supervised action alignment and segmentation","authors":"Chien-Yi Chang, De-An Huang, Yanan Sui, Li Fei-Fei, Juan Carlos Niebles","year":"2019","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dietterich, 2002] Thomas G Dietterich. Ensemble learn- ing. The handbook of brain theory and neural networks, 2(1):110-125, 2002.","title":"Ensemble learning","authors":"G Thomas, Dietterich","year":"2002","venue":"The handbook of brain theory and neural networks","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Du et al., 2022] Dazhao Du, Bing Su, Yu Li, Zhongang Qi, Lingyu Si, and Ying Shan. Do we really need tempo- ral convolutions in action segmentation? arXiv preprint arXiv:2205.13425, 2022.","title":"Do we really need temporal convolutions in action segmentation","authors":"Dazhao Du, Bing Su, Yu Li, Zhongang Qi, Lingyu Si, Ying Shan","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Farha and Gall, 2019] Yazan Abu Farha and Jurgen Gall. Ms-tcn: Multi-stage temporal convolutional network for action segmentation. In Proceedings of the IEEE Confer- ence on Computer Vision and Pattern Recognition, pages 3575-3584, 2019.","title":"Ms-tcn: Multi-stage temporal convolutional network for action segmentation","authors":"Farha, Abu Gall ; Yazan, Jurgen Farha, Gall","year":"2019","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Fathi et al., 2011] Alireza Fathi, Xiaofeng Ren, and James M Rehg. Learning to recognize objects in egocen- tric activities. In CVPR 2011, pages 3281-3288. IEEE, 2011.","title":"Learning to recognize objects in egocentric activities","authors":"Alireza Fathi, Xiaofeng Ren, James M Rehg","year":"2011","venue":"CVPR 2011","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Fayyaz and Gall, 2020] Mohsen Fayyaz and Jurgen Gall. Sct: Set constrained temporal transformer for set su- pervised action segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 501-510, 2020.","title":"Sct: Set constrained temporal transformer for set supervised action segmentation","authors":"Mohsen Fayyaz, Jurgen Gall","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Han and Kamber, 2006] Jiawei Han and Micheline Kamber. Data mining: concepts and techniques, 2nd. University of Illinois at Urbana Champaign: Morgan Kaufmann, 2006.","title":"Data mining: concepts and techniques","authors":"Kamber ; Han, Jiawei Han, Micheline Kamber","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Huang et al., 2016] De-An Huang, Li Fei-Fei, and Juan Carlos Niebles. Connectionist temporal model- ing for weakly supervised action labeling. In European Conference on Computer Vision, pages 137-153. Springer, 2016.","title":"Connectionist temporal modeling for weakly supervised action labeling","authors":"De-An, Li Huang, Juan Carlos Fei-Fei, Niebles","year":"2016","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Iscen et al., 2019] Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondrej Chum. Label propagation for deep semi-supervised learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5070-5079, 2019.","title":"Label propagation for deep semi-supervised learning","authors":"Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, Ondrej Chum","year":"2019","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Ishikawa et al., 2021] Yuchi Ishikawa, Seito Kasai, Yoshimitsu Aoki, and Hirokatsu Kataoka. Alleviating over-segmentation errors by detecting action boundaries. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2322-2331, 2021.","title":"Alleviating over-segmentation errors by detecting action boundaries","authors":"Ishikawa","year":"2021","venue":"Proceedings of the IEEE/CVF winter conference on applications of computer vision","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Khan et al., 2022] Hamza Khan, Sanjay Haresh, Awais Ahmed, Shakeeb Siddiqui, Andrey Konin, M Zeeshan Zia, and Quoc-Huy Tran. Timestamp-supervised ac- tion segmentation with graph convolutional networks. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2022.","title":"Timestamp-supervised action segmentation with graph convolutional networks","authors":"Hamza Khan, Sanjay Haresh, Awais Ahmed, Shakeeb Siddiqui, Andrey Konin, M Zeeshan, Zia, Quoc-Huy Tran","year":"2022","venue":"IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kingma and Ba, 2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Interna- tional Conference on Learning Representations, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kuehne et al., 2014] Hilde Kuehne, Ali Arslan, and Thomas Serre. The language of actions: Recovering the syntax and semantics of goal-directed human activities. In CVPR, pages 780-787, 2014.","title":"The language of actions: Recovering the syntax and semantics of goal-directed human activities","authors":"Hilde Kuehne, Ali Arslan, Thomas Serre","year":"2014","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kuehne et al., 2016] Hilde Kuehne, Juergen Gall, and Thomas Serre. An end-to-end generative framework for video segmentation and recognition. In 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1-8. IEEE, 2016.","title":"An end-to-end generative framework for video segmentation and recognition","authors":"Hilde Kuehne, Juergen Gall, Thomas Serre","year":"2016","venue":"IEEE Winter Conference on Applications of Computer Vision (WACV)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kuehne et al., 2017] Hilde Kuehne, Alexander Richard, and Juergen Gall. Weakly supervised learning of actions from transcripts. Computer Vision and Image Understanding, 163:78-89, 2017.","title":"Weakly supervised learning of actions from transcripts","authors":"Kuehne","year":"2017","venue":"Computer Vision and Image Understanding","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lea et al., 2017] Colin Lea, Michael D Flynn, Rene Vidal, Austin Reiter, and Gregory D Hager. Temporal convolu- tional networks for action segmentation and detection. In CVPR, pages 156-165, 2017.","title":"Temporal convolutional networks for action segmentation and detection","authors":"Colin Lea, Rene Michael D Flynn, Austin Vidal, Gregory D Reiter, Hager","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lee and others, 2013] Dong-Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, page 896, 2013.","title":"Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks","authors":"Dong-Hyun Lee","year":"2013","venue":"Workshop on challenges in representation learning","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Li and Todorovic, 2021] Jun Li and Sinisa Todorovic. Anchor-constrained viterbi for set-supervised action seg- mentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9806- 9815, 2021.","title":"Anchor-constrained viterbi for set-supervised action segmentation","authors":"Li, Todorovic, Jun Li, Sinisa Todorovic","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Li et al., 2019] Jun Li, Peng Lei, and Sinisa Todorovic. Weakly supervised energy-based learning for action seg- mentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6243-6251, 2019.","title":"Weakly supervised energy-based learning for action segmentation","authors":"Jun Li, Peng Lei, Sinisa Todorovic","year":"2019","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Li et al., 2020] Shi-Jie Li, Yazan AbuFarha, Yun Liu, Ming- Ming Cheng, and Juergen Gall. Ms-tcn++: Multi-stage temporal convolutional network for action segmentation. IEEE Transactions on Pattern Analysis and Machine In- telligence, 2020.","title":"Ms-tcn++: Multi-stage temporal convolutional network for action segmentation","authors":"Shi-Jie Li, Yazan Abufarha, Yun Liu, Ming-Ming Cheng, Juergen Gall","year":"2020","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Li et al., 2021] Zhe Li, Yazan Abu Farha, and Jurgen Gall. Temporal action segmentation from timestamp supervi- sion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8365- 8374, 2021.","title":"Temporal action segmentation from timestamp supervision","authors":"Zhe Li, Yazan Abu Farha, Jurgen Gall","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Qiang et al., 2023] Wenwen Qiang, Jiangmeng Li, Bing Su, Jianlong Fu, Hui Xiong, and Wen Jirong. Meta attention- generation network for cross-granularity few-shot learn- ing. International Journal of Computer Vision, 2023.","title":"Meta attentiongeneration network for cross-granularity few-shot learning","authors":"Wenwen Qiang, Jiangmeng Li, Bing Su, Jianlong Fu, Hui Xiong, Wen Jirong","year":"2023","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Rahaman et al., 2022] Rahul Rahaman, Dipika Singhania, Alexandre Thiery, and Angela Yao. A generalized and robust framework for timestamp supervision in temporal action segmentation. In ECCV, pages 279-296. Springer, 2022.","title":"A generalized and robust framework for timestamp supervision in temporal action segmentation","authors":"Dipika Rahul Rahaman, Alexandre Singhania, Angela Thiery, Yao","year":"2022","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Richard et al., 2018a] Alexander Richard, Hilde Kuehne, and Juergen Gall. Action sets: Weakly supervised action segmentation without ordering constraints. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5987-5996, 2018.","title":"Action sets: Weakly supervised action segmentation without ordering constraints","authors":"Richard","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Richard et al., 2018b] Alexander Richard, Hilde Kuehne, Ahsan Iqbal, and Juergen Gall. Neuralnetwork-viterbi: A framework for weakly supervised video learning. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7386-7395, 2018.","title":"Neuralnetwork-viterbi: A framework for weakly supervised video learning","authors":"Richard","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Singh et al., 2016] Bharat Singh, Tim K Marks, Michael Jones, Oncel Tuzel, and Ming Shao. A multi-stream bi- directional recurrent neural network for fine-grained ac- tion detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1961- 1970, 2016.","title":"A multi-stream bidirectional recurrent neural network for fine-grained action detection","authors":"Bharat Singh, Tim K Marks, Michael Jones, Oncel Tuzel, Ming Shao","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Stein and McKenna, 2013] Sebastian Stein and Stephen J McKenna. Combining embedded accelerometers with computer vision for recognizing food preparation activi- ties. In Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing, pages 729-738, 2013.","title":"Combining embedded accelerometers with computer vision for recognizing food preparation activities","authors":"Mckenna Stein, Sebastian Stein, Stephen J Mckenna","year":"2013","venue":"Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Tang et al., 2012] Kevin Tang, Li Fei-Fei, and Daphne Koller. Learning latent temporal structure for complex event detection. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 1250-1257. IEEE, 2012.","title":"Learning latent temporal structure for complex event detection","authors":"Kevin Tang, Li Fei-Fei, Daphne Koller","year":"2012","venue":"2012 IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wang et al., 2020] Zhenzhi Wang, Ziteng Gao, Limin Wang, Zhifeng Li, and Gangshan Wu. Boundary-aware cascade networks for temporal action segmentation. In ECCV, pages 34-51. Springer, 2020.","title":"Boundary-aware cascade networks for temporal action segmentation","authors":"Wang","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Yang et al., 2017] Bo Yang, Xiao Fu, Nicholas D Sidiropoulos, and Mingyi Hong. Towards k-means- friendly spaces: Simultaneous deep learning and clustering. In international conference on machine learning, pages 3861-3870. PMLR, 2017.","title":"Towards k-meansfriendly spaces: Simultaneous deep learning and clustering","authors":"Bo Yang, Xiao Fu, Nicholas D Sidiropoulos, Mingyi Hong","year":"2017","venue":"international conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Yi et al., 2021] Fangqiu Yi, Hongyu Wen, and Tingting Jiang. Asformer: Transformer for action segmentation. In The British Machine Vision Conference, 2021.","title":"Asformer: Transformer for action segmentation","authors":"Yi","year":"2021","venue":"The British Machine Vision Conference","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}