{"type":"ScholarlyArticle","title":"HOI-aware Adaptive Network for Weakly-supervised Action Segmentation","authors":[{"name":"Runzhong Zhang","slug":"runzhong-zhang","given":"Runzhong","family":"Zhang","affiliation":"Nanyang Technological University"},{"name":"Suchen Wang","slug":"suchen-wang","given":"Suchen","family":"Wang","affiliation":"Nanyang Technological University"},{"name":"Yueqi Duan","slug":"yueqi-duan","given":"Yueqi","family":"Duan","affiliation":"Tsinghua University"},{"name":"Yansong Tang","slug":"yansong-tang","given":"Yansong","family":"Tang","affiliation":"Tsinghua University"},{"name":"Yue Zhang","slug":"yue-zhang","given":"Yue","family":"Zhang","affiliation":"Beijing Jiaotong University"},{"name":"Yap-Peng Tan","slug":"yap-peng-tan","given":"Yap-Peng","family":"Tan","affiliation":"Nanyang Technological University"}],"abstract":"In this paper, we propose an HOI-aware adaptive network named AdaAct for weakly-supervised action segmentation. Most existing methods learn a fixed network to predict the action of each frame with the neighboring frames. However, this would result in ambiguity when estimating similar actions, such as pouring juice and pouring coffee. To address this, we aim to exploit temporally global but spatially local human-object interactions (HOI) as video-level prior knowledge for action segmentation. The long-term HOI sequence provides crucial contextual information to distinguish ambiguous actions, where our network dynamically adapts to the given HOI sequence at test time. More specifically, we first design a video HOI encoder that extracts, selects, and integrates the most representative HOI throughout the video. Then, we propose a two-branch HyperNetwork to learn an adaptive temporal encoder, which automatically adjusts the parameters based on the HOI information of various videos on the fly. Extensive experiments on two widely-used datasets including Breakfast and 50Salads demonstrate the effectiveness of our method under different evaluation metrics.","doi":"10.24963/ijcai.2023/191","url":"https://ijcai.org/proceedings/2023/191","pdf":"https://ijcai.org/proceedings/2023/0191.pdf","markdown":"https://ijcai.org/proceedings/2023/191.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/191","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Computer Vision","track":"Main Track","number":191,"pages":{"first":1722,"last":1730},"keywords":["Computer Vision: CV: Video analysis and understanding","Computer Vision: CV: Action and behavior recognition"],"videos":[],"references":[{"position":1,"raw":"[Bearman et al., 2016] Amy Bearman, Olga Russakovsky, Vittorio Ferrari, and Li Fei-Fei. What's the point: Seman- tic segmentation with point supervision. In ECCV, pages 549-565, 2016.","title":"What's the point: Semantic segmentation with point supervision","authors":"Amy Bearman, Olga Russakovsky, Vittorio Ferrari, Li Fei-Fei","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bojanowski et al., 2014] Piotr Bojanowski, Rémi Lajugie, Francis Bach, Ivan Laptev, Jean Ponce, Cordelia Schmid, and Josef Sivic. Weakly supervised action labeling in videos under ordering constraints. In ECCV, pages 628- 643, 2014.","title":"Weakly supervised action labeling in videos under ordering constraints","authors":"Piotr Bojanowski, Rémi Lajugie, Francis Bach, Ivan Laptev, Jean Ponce, Cordelia Schmid, Josef Sivic","year":"2014","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chang et al., 2019] Chien-Yi Chang, De-An Huang, Yanan Sui, Li Fei-Fei, and Juan Carlos Niebles. D3tw: Discrimi- native differentiable dynamic time warping for weakly su- pervised action alignment and segmentation. In CVPR, pages 3546-3555, 2019.","title":"D3tw: Discriminative differentiable dynamic time warping for weakly supervised action alignment and segmentation","authors":"Chien-Yi Chang, De-An Huang, Yanan Sui, Li Fei-Fei, Juan Carlos Niebles","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2021] Mingfei Chen, Yue Liao, Si Liu, Zhiyuan Chen, Fei Wang, and Chen Qian. Reformulating hoi de- tection as adaptive set prediction. In CVPR, pages 9004- 9013, 2021.","title":"Reformulating hoi detection as adaptive set prediction","authors":"Mingfei Chen, Yue Liao, Si Liu, Zhiyuan Chen, Fei Wang, Chen Qian","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chung et al., 2014] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evalua- tion of gated recurrent neural networks on sequence mod- eling. arXiv preprint arXiv:1412.3555, 2014.","title":"Empirical evaluation of gated recurrent neural networks on sequence modeling","authors":"Chung","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Ding and Xu, 2018] Li Ding and Chenliang Xu. Weakly- supervised action segmentation with iterative soft bound- ary assignment. In CVPR, pages 6508-6516, 2018.","title":"Weaklysupervised action segmentation with iterative soft boundary assignment","authors":"Xu ; ] Ding, Ding Li, Chenliang Xu","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Fang et al., 2018] Hao-Shu Fang, Jinkun Cao, Yu-Wing Tai, and Cewu Lu. Pairwise body-part attention for recognizing human-object interactions. In ECCV, pages 51-67, 2018.","title":"Pairwise body-part attention for recognizing human-object interactions","authors":"Jinkun Hao-Shu Fang, Yu-Wing Cao, Cewu Tai, Lu","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Farha and Gall, 2019] Yazan Abu Farha and Jurgen Gall. Ms-tcn: Multi-stage temporal convolutional network for action segmentation. In CVPR, pages 3575-3584, 2019.","title":"Ms-tcn: Multi-stage temporal convolutional network for action segmentation","authors":"Farha, Abu Gall ; Yazan, Jurgen Farha, Gall","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Fayyaz and Gall, 2020] Mohsen Fayyaz and Jurgen Gall. Sct: Set constrained temporal transformer for set super- vised action segmentation. In CVPR, pages 501-510, 2020.","title":"Sct: Set constrained temporal transformer for set supervised action segmentation","authors":"Mohsen Fayyaz, Jurgen Gall","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ha et al., 2016] David Ha, Andrew Dai, and Quoc V Le. Hypernetworks. arXiv preprint arXiv:1609.09106, 2016.","title":"Hypernetworks","authors":"David Ha, Andrew Dai, Quoc V Le","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Huang et al., 2016] De-An Huang, Li Fei-Fei, and Juan Carlos Niebles. Connectionist temporal model- ing for weakly supervised action labeling. In ECCV, pages 137-153, 2016.","title":"Connectionist temporal modeling for weakly supervised action labeling","authors":"De-An, Li Huang, Juan Carlos Fei-Fei, Niebles","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Huang et al., 2020] Yifei Huang, Yusuke Sugano, and Yoichi Sato. Improving action segmentation via graph- based temporal reasoning. In CVPR, pages 14024-14034, 2020.","title":"Improving action segmentation via graphbased temporal reasoning","authors":"Huang","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Karaman et al., 2014] Svebor Karaman, Lorenzo Seidenari, and Alberto Del Bimbo. Fast saliency based pooling of fisher encoded dense trajectories. In ECCV THUMOS Workshop, volume 1, page 5, 2014.","title":"Fast saliency based pooling of fisher encoded dense trajectories","authors":"Karaman","year":"2014","venue":"ECCV THUMOS Workshop","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kim et al., 2020] Bumsoo Kim, Taeho Choi, Jaewoo Kang, and Hyunwoo J Kim. Uniondet: Union-level detector towards real-time human-object interaction detection. In ECCV, pages 498-514, 2020.","title":"Uniondet: Union-level detector towards real-time human-object interaction detection","authors":"Kim","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kuehne et al., 2014] Hilde Kuehne, Ali Arslan, and Thomas Serre. The language of actions: Recovering the syntax and semantics of goal-directed human activities. In CVPR, pages 780-787, 2014.","title":"The language of actions: Recovering the syntax and semantics of goal-directed human activities","authors":"Hilde Kuehne, Ali Arslan, Thomas Serre","year":"2014","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kuehne et al., 2016] Hilde Kuehne, Juergen Gall, and Thomas Serre. An end-to-end generative framework for video segmentation and recognition. In WACV, pages 1-8, 2016.","title":"An end-to-end generative framework for video segmentation and recognition","authors":"Hilde Kuehne, Juergen Gall, Thomas Serre","year":"2016","venue":"WACV","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kuehne et al., 2017] Hilde Kuehne, Alexander Richard, and Juergen Gall. Weakly supervised learning of actions from transcripts. Computer Vision and Image Understanding, 163:78-89, 2017.","title":"Weakly supervised learning of actions from transcripts","authors":"Kuehne","year":"2017","venue":"Computer Vision and Image Understanding","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lea et al., 2017] Colin Lea, Michael D Flynn, Rene Vidal, Austin Reiter, and Gregory D Hager. Temporal convolu- tional networks for action segmentation and detection. In CVPR, pages 156-165, 2017.","title":"Temporal convolutional networks for action segmentation and detection","authors":"Colin Lea, Rene Michael D Flynn, Austin Vidal, Gregory D Reiter, Hager","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Lei and Todorovic, 2018] Peng Lei and Sinisa Todorovic. Temporal deformable residual networks for action seg- mentation in videos. In CVPR, pages 6742-6751, 2018.","title":"Temporal deformable residual networks for action segmentation in videos","authors":"Todorovic ; ] Lei, Lei Peng, Sinisa Todorovic","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Li and Todorovic, 2020] Jun Li and Sinisa Todorovic. Set- constrained viterbi for set-supervised action segmentation. In CVPR, pages 10820-10829, 2020.","title":"Setconstrained viterbi for set-supervised action segmentation","authors":"Li, Todorovic, Jun Li, Sinisa Todorovic","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Li et al., 2019a] Jun Li, Peng Lei, and Sinisa Todorovic. Weakly supervised energy-based learning for action seg- mentation. In ICCV, pages 6243-6251, 2019.","title":"Weakly supervised energy-based learning for action segmentation","authors":"Li","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Li et al., 2019b] Yong-Lu Li, Siyuan Zhou, Xijie Huang, Liang Xu, Ze Ma, Hao-Shu Fang, Yanfeng Wang, and Cewu Lu. Transferable interactiveness knowledge for human-object interaction detection. In CVPR, pages 3585-3594, 2019.","title":"Transferable interactiveness knowledge for human-object interaction detection","authors":"Li","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Li et al., 2020] Shi-Jie Li, Yazan AbuFarha, Yun Liu, Ming- Ming Cheng, and Juergen Gall. Ms-tcn++: Multi-stage temporal convolutional network for action segmentation. TPAMI, 2020.","title":"Ms-tcn++: Multi-stage temporal convolutional network for action segmentation","authors":"Shi-Jie Li, Yazan Abufarha, Yun Liu, Ming-Ming Cheng, Juergen Gall","year":"2020","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Li et al., 2021] Zhe Li, Yazan Abu Farha, and Jurgen Gall. Temporal action segmentation from timestamp supervi- sion. In CVPR, pages 8365-8374, 2021.","title":"Temporal action segmentation from timestamp supervision","authors":"Zhe Li, Yazan Abu Farha, Jurgen Gall","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Li et al., 2022] Muheng Li, Lei Chen, Yueqi Duan, Zhilan Hu, Jianjiang Feng, Jie Zhou, and Jiwen Lu. Bridge- prompt: Towards ordinal action understanding in instruc- tional videos. In CVPR, pages 19880-19889, 2022.","title":"Bridgeprompt: Towards ordinal action understanding in instructional videos","authors":"Muheng Li, Lei Chen, Yueqi Duan, Zhilan Hu, Jianjiang Feng, Jie Zhou, Jiwen Lu","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Liao et al., 2020] Yue Liao, Si Liu, Fei Wang, Yanjie Chen, Chen Qian, and Jiashi Feng. Ppdm: Parallel point detec- tion and matching for real-time human-object interaction detection. In CVPR, pages 482-490, 2020.","title":"Ppdm: Parallel point detection and matching for real-time human-object interaction detection","authors":"Yue Liao, Si Liu, Fei Wang, Yanjie Chen, Chen Qian, Jiashi Feng","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Lu and Elhamifar, 2021] Zijia Lu and Ehsan Elhamifar. Weakly-supervised action segmentation and alignment via transcript-aware union-of-subspaces learning. In ICCV, pages 8085-8095, 2021.","title":"Weakly-supervised action segmentation and alignment via transcript-aware union-of-subspaces learning","authors":"Elhamifar ; Lu, Zijia Lu, Ehsan Elhamifar","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Neubeck and Van Gool, 2006] Alexander Neubeck and Luc Van Gool. Efficient non-maximum suppression. In ICPR, volume 3, pages 850-855, 2006.","title":"Efficient non-maximum suppression","authors":"Van Neubeck, Alexander Gool, Luc Neubeck, Van Gool","year":"2006","venue":"ICPR","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Park et al., 2022] Junyong Park, Daekyum Kim, Sejoon Huh, and Sungho Jo. Maximization and restoration: Ac- tion segmentation through dilation passing and temporal reconstruction. Pattern Recognition, 129:108764, 2022.","title":"Maximization and restoration: Action segmentation through dilation passing and temporal reconstruction","authors":"Junyong Park, Daekyum Kim, Sejoon Huh, Sungho Jo","year":"2022","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Pirsiavash and Ramanan, 2014] Hamed Pirsiavash and Deva Ramanan. Parsing videos of actions with segmental grammars. In CVPR, pages 612-619, 2014.","title":"Parsing videos of actions with segmental grammars","authors":"Ramanan Pirsiavash, Hamed Pirsiavash, Deva Ramanan","year":"2014","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Qi et al., 2018] Siyuan Qi, Wenguan Wang, Baoxiong Jia, Jianbing Shen, and Song-Chun Zhu. Learning human- object interactions by graph parsing neural networks. In ECCV, pages 401-417, 2018.","title":"Learning humanobject interactions by graph parsing neural networks","authors":"Siyuan Qi, Wenguan Wang, Baoxiong Jia, Jianbing Shen, Song-Chun Zhu","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Richard et al., 2017] Alexander Richard, Hilde Kuehne, and Juergen Gall. Weakly supervised action learning with rnn based fine-to-coarse modeling. In CVPR, pages 754- 763, 2017.","title":"Weakly supervised action learning with rnn based fine-to-coarse modeling","authors":"Alexander Richard, Hilde Kuehne, Juergen Gall","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Richard et al., 2018] Alexander Richard, Hilde Kuehne, Ahsan Iqbal, and Juergen Gall. Neuralnetwork-viterbi: A framework for weakly supervised video learning. In CVPR, pages 7386-7395, 2018.","title":"Neuralnetwork-viterbi: A framework for weakly supervised video learning","authors":"Alexander Richard, Hilde Kuehne, Ahsan Iqbal, Juergen Gall","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Rohrbach et al., 2012] Marcus Rohrbach, Sikandar Amin, Mykhaylo Andriluka, and Bernt Schiele. A database for fine grained activity detection of cooking activities. In CVPR, pages 1194-1201, 2012.","title":"A database for fine grained activity detection of cooking activities","authors":"Marcus Rohrbach, Sikandar Amin, Mykhaylo Andriluka, Bernt Schiele","year":"2012","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Shan et al., 2020] Dandan Shan, Jiaqi Geng, Michelle Shu, and David F Fouhey. Understanding human hands in con- tact at internet scale. In CVPR, pages 9869-9878, 2020.","title":"Understanding human hands in contact at internet scale","authors":"Dandan Shan, Jiaqi Geng, Michelle Shu, David F Fouhey","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Singh et al., 2016] Bharat Singh, Tim K Marks, Michael Jones, Oncel Tuzel, and Ming Shao. A multi-stream bi- directional recurrent neural network for fine-grained ac- tion detection. In CVPR, pages 1961-1970, 2016.","title":"A multi-stream bidirectional recurrent neural network for fine-grained action detection","authors":"Bharat Singh, Tim K Marks, Michael Jones, Oncel Tuzel, Ming Shao","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Souri et al., 2021] Yaser Souri, Mohsen Fayyaz, Luca Min- ciullo, Gianpiero Francesca, and Juergen Gall. Fast weakly supervised action segmentation using mutual consistency. TPAMI, 2021.","title":"Fast weakly supervised action segmentation using mutual consistency","authors":"Yaser Souri, Mohsen Fayyaz, Luca Minciullo, Gianpiero Francesca, Juergen Gall","year":"2021","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Stein and McKenna, 2013] Sebastian Stein and Stephen J McKenna. Combining embedded accelerometers with computer vision for recognizing food preparation activi- ties. In UbiComp, pages 729-738, 2013.","title":"Combining embedded accelerometers with computer vision for recognizing food preparation activities","authors":"Mckenna Stein, Sebastian Stein, Stephen J Mckenna","year":"2013","venue":"UbiComp","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Ulutan et al., 2020] Oytun Ulutan, ASM Iftekhar, and Ban- galore S Manjunath. Vsgnet: Spatial attention network for detecting human object interactions using graph convolu- tions. In CVPR, pages 13617-13626, 2020.","title":"Oytun Ulutan, ASM Iftekhar, and Bangalore S Manjunath. Vsgnet: Spatial attention network for detecting human object interactions using graph convolutions","authors":"Ulutan","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Wang et al., 2020a] Tiancai Wang, Tong Yang, Martin Danelljan, Fahad Shahbaz Khan, Xiangyu Zhang, and Jian Sun. Learning human-object interaction detection using interaction points. In CVPR, pages 4116-4125, 2020.","title":"Learning human-object interaction detection using interaction points","authors":"Wang","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Wang et al., 2020b] Zhenzhi Wang, Ziteng Gao, Limin Wang, Zhifeng Li, and Gangshan Wu. Boundary-aware cascade networks for temporal action segmentation. In ECCV, pages 34-51, 2020.","title":"Boundary-aware cascade networks for temporal action segmentation","authors":"Wang","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Yi et al., 2021] Fangqiu Yi, Hongyu Wen, and Tingting Jiang. Asformer: Transformer for action segmentation. arXiv preprint arXiv:2110.08568, 2021.","title":"Asformer: Transformer for action segmentation","authors":"Yi","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Zhang et al., 2021] Frederic Z Zhang, Dylan Campbell, and Stephen Gould. Spatially conditioned graphs for detecting human-object interactions. In ICCV, pages 13319-13327, 2021.","title":"Spatially conditioned graphs for detecting human-object interactions","authors":"Frederic Z Zhang, Dylan Campbell, Stephen Gould","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":45,"raw":"[Zhou and Chi, 2019] Penghao Zhou and Mingmin Chi. Re- lation parsing neural network for human-object interaction detection. In ICCV, pages 843-851, 2019.","title":"Penghao Zhou and Mingmin Chi. Relation parsing neural network for human-object interaction detection","authors":"Chi Zhou","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":46,"raw":"[Zhou et al., 2020] Tianfei Zhou, Wenguan Wang, Siyuan Qi, Haibin Ling, and Jianbing Shen. Cascaded human- object interaction recognition. In CVPR, pages 4263- 4272, 2020.","title":"Cascaded humanobject interaction recognition","authors":"Zhou","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}