{"type":"ScholarlyArticle","title":"CoFInAl: Enhancing Action Quality Assessment with Coarse-to-Fine Instruction Alignment","authors":[{"name":"Kanglei Zhou","slug":"kanglei-zhou","given":"Kanglei","family":"Zhou","affiliation":"State Key Laboratory of Virtual Reality Technology and Systems , Beihang University"},{"name":"Junlin Li","slug":"junlin-li","given":"Junlin","family":"Li","affiliation":"China Three Gorges University"},{"name":"Ruizhi Cai","slug":"ruizhi-cai","given":"Ruizhi","family":"Cai","affiliation":"State Key Laboratory of Virtual Reality Technology and Systems , Beihang University"},{"name":"Liyuan Wang","slug":"liyuan-wang","given":"Liyuan","family":"Wang","affiliation":"Department of Computer Science and Technology , Institute for AI , BNRist Center , Tsinghua-Bosch Joint ML Center , THBI Lab , Tsinghua University"},{"name":"Xingxing Zhang","slug":"xingxing-zhang","given":"Xingxing","family":"Zhang","affiliation":"Department of Computer Science and Technology , Institute for AI , BNRist Center , Tsinghua-Bosch Joint ML Center , THBI Lab , Tsinghua University"},{"name":"Xiaohui Liang","slug":"xiaohui-liang","given":"Xiaohui","family":"Liang","affiliation":"State Key Laboratory of Virtual Reality Technology and Systems , Beihang University; Zhongguancun Laboratory"}],"abstract":"Action Quality Assessment (AQA) is pivotal for quantifying actions across domains like sports and medical care. Existing methods often rely on pre-trained backbones from large-scale action recognition datasets to boost performance on smaller AQA datasets. However, this common strategy yields suboptimal results due to the inherent struggle of these backbones to capture the subtle cues essential for AQA. Moreover, fine-tuning on smaller datasets risks overfitting. To address these issues, we propose Coarse-to-Fine Instruction Alignment (CoFInAl). Inspired by recent advances in large language model tuning, CoFInAl aligns AQA with broader pre-trained tasks by reformulating it as a coarse-to-fine classification task. Initially, it learns grade prototypes for coarse assessment and then utilizes fixed sub-grade prototypes for fine-grained assessment. This hierarchical approach mirrors the judging process, enhancing interpretability within the AQA framework. Experimental results on two long-term AQA datasets demonstrate CoFInAl achieves state-of-the-art performance with significant correlation gains of 5.49% and 3.55% on Rhythmic Gymnastics and Fis-V, respectively. Our Code is available at https://github.com/ZhouKanglei/CoFInAl_AQA.","doi":"10.24963/ijcai.2024/196","url":"https://ijcai.org/proceedings/2024/196","pdf":"https://ijcai.org/proceedings/2024/0196.pdf","markdown":"https://ijcai.org/proceedings/2024/196.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/196","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Computer Vision","track":"Main Track","number":196,"pages":{"first":1771,"last":1779},"keywords":["Computer Vision: CV: Video analysis and understanding","Computer Vision: CV: Action and behavior recognition"],"videos":[],"references":[{"position":1,"raw":"[Bai et al., 2022] Yang Bai, Desen Zhou, Songyang Zhang, Jian Wang, Errui Ding, Yu Guan, Yang Long, and Jing- dong Wang. Action quality assessment with temporal pars- ing transformer. In ECCV, pages 422-438, 2022.","title":"Action quality assessment with temporal parsing transformer","authors":"Yang Bai, Desen Zhou, Songyang Zhang, Jian Wang, Errui Ding, Yu Guan, Yang Long, Jingdong Wang","year":"2022","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Carreira and Zisserman, 2017] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In CVPR, pages 6299-6308, 2017.","title":"Quo vadis, action recognition? a new model and the kinetics dataset","authors":"Joao Carreira, Andrew Zisserman","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Dadashzadeh et al., 2024] Amirhossein Dadashzadeh, Shuchao Duan, Alan Whone, and Majid Mirmehdi. Pecop: Parameter efficient continual pretraining for action quality assessment. In WACV, pages 42-52, 2024.","title":"Pecop: Parameter efficient continual pretraining for action quality assessment","authors":"Dadashzadeh","year":"2024","venue":"WACV","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Ding et al., 2023] Xinpeng Ding, Xiaowei Xu, and Xi- aomeng Li. Sedskill: Surgical events driven method for skill assessment from thoracoscopic surgical videos. In MICCAI, pages 35-45, 2023.","title":"Sedskill: Surgical events driven method for skill assessment from thoracoscopic surgical videos","authors":"Ding","year":"2023","venue":"MICCAI","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Fang et al., 2021] Cong Fang, Hangfeng He, Qi Long, and Weijie J Su. Exploring deep neural networks via layer- peeled model: Minority collapse in imbalanced training. PNAS, 118(43):e2103091118, 2021.","title":"Exploring deep neural networks via layerpeeled model: Minority collapse in imbalanced training","authors":"Cong Fang, Hangfeng He, Qi Long, Weijie J Su","year":"2021","venue":"PNAS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Galanti et al., 2021] Tomer Galanti, András György, and Marcus Hutter. On the role of neural collapse in transfer learning. arXiv preprint arXiv:2112.15121, 2021.","title":"On the role of neural collapse in transfer learning","authors":"Tomer Galanti, András György, Marcus Hutter","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Graf et al., 2021] Florian Graf, Christoph Hofer, Marc Ni- ethammer, and Roland Kwitt. Dissecting supervised con- trastive learning. In ICML, pages 3821-3830, 2021.","title":"Dissecting supervised contrastive learning","authors":"Florian Graf, Christoph Hofer, Marc Niethammer, Roland Kwitt","year":"2021","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Han et al., 2021] XY Han, Vardan Papyan, and David L Donoho. Neural collapse under mse loss: Proximity to and dynamics on the central path. arXiv preprint arXiv:2106.02073, 2021.","title":"Neural collapse under mse loss: Proximity to and dynamics on the central path","authors":"Han","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kay et al., 2017] Will Kay, Joao Carreira, Karen Si- monyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv:1705.06950, 2017.","title":"The kinetics human action video dataset","authors":"Kay","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Liu et al., 2022] Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin trans- former. In CVPR, pages 3202-3211, 2022.","title":"Video swin transformer","authors":"Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, Han Hu","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Liu et al., 2023] Yanchao Liu, Xina Cheng, and Takeshi Ikenaga. A figure skating jumping dataset for replay- guided action quality assessment. In ACM MM, pages 2437-2445, 2023.","title":"A figure skating jumping dataset for replayguided action quality assessment","authors":"Liu","year":"2023","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Pan et al., 2019] Jia-Hui Pan, Jibin Gao, and Wei-Shi Zheng. Action assessment by joint relation graphs. In ICCV, pages 6331-6340, 2019.","title":"Action assessment by joint relation graphs","authors":"Jia-Hui Pan, Jibin Gao, Wei-Shi Zheng","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Papyan et al., 2020] Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. PNAS, 117(40):24652- 24663, 2020.","title":"Prevalence of neural collapse during the terminal phase of deep learning training","authors":"Papyan","year":"2020","venue":"PNAS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Parmar and Morris, 2019a] Paritosh Parmar and Brendan Morris. Action quality assessment across multiple actions. In WACV, pages 1468-1476, 2019.","title":"Action quality assessment across multiple actions","authors":"Morris ; Parmar, Paritosh Parmar, Brendan Morris","year":"2019","venue":"WACV","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Parmar and Morris, 2019b] Paritosh Parmar and Bren- dan Tran Morris. What and how well you performed? a multitask learning approach to action quality assessment. In CVPR, pages 304-313, 2019.","title":"What and how well you performed? a multitask learning approach to action quality assessment","authors":"Morris ; Parmar, Paritosh Parmar, Brendan Tran, Morris","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Parmar and Tran Morris, 2017] Paritosh Parmar and Bren- dan Tran Morris. Learning to score olympic events. In CVPR workshops, pages 20-28, 2017.","title":"Learning to score olympic events","authors":"Tran Parmar, ; Paritosh Morris, Brendan Parmar, Morris Tran","year":"2017","venue":"CVPR workshops","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Pirsiavash et al., 2014] Hamed Pirsiavash, Carl Vondrick, and Antonio Torralba. Assessing the quality of actions. In ECCV, pages 556-571, 2014.","title":"Assessing the quality of actions","authors":"Hamed Pirsiavash, Carl Vondrick, Antonio Torralba","year":"2014","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Smith et al., 2020] Eric Michael Smith, Mary Williamson, Kurt Shuster, Jason Weston, and Y-Lan Boureau. Can you put it all together: Evaluating conversational agents' ability to blend skills. In Proceedings of the 58th An- nual Meeting of the Association for Computational Lin- guistics, pages 2021-2030, Online, July 2020. Association for Computational Linguistics.","title":"Can you put it all together: Evaluating conversational agents' ability to blend skills","authors":"Eric Michael, Smith, Mary Williamson, Kurt Shuster, Jason Weston, Y-Lan Boureau","year":"2020","venue":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Tang et al., 2020] Yansong Tang, Zanlin Ni, Jiahuan Zhou, Danyang Zhang, Jiwen Lu, Ying Wu, and Jie Zhou. Uncertainty-aware score distribution learning for action quality assessment. In CVPR, pages 9839-9848, 2020.","title":"Uncertainty-aware score distribution learning for action quality assessment","authors":"Yansong Tang, Zanlin Ni, Jiahuan Zhou, Danyang Zhang, Jiwen Lu, Ying Wu, Jie Zhou","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Tirer and Bruna, 2022] Tom Tirer and Joan Bruna. Ex- tended unconstrained features model for exploring deep neural collapse. In ICML, pages 21478-21505, 2022.","title":"Extended unconstrained features model for exploring deep neural collapse","authors":"Bruna ; Tirer, Tom Tirer, Joan Bruna","year":"2022","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Tran et al., 2015] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spa- tiotemporal features with 3d convolutional networks. In ICCV, pages 4489-4497, 2015.","title":"Learning spatiotemporal features with 3d convolutional networks","authors":"Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, Manohar Paluri","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Van der Maaten and Hinton, 2008] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.","title":"Visualizing data using t-sne","authors":"Van Der Maaten, Laurens Hinton, Geoffrey Van Der Maaten, Hinton","year":"2008","venue":"Journal of machine learning research","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2020] Tianyu Wang, Yijie Wang, and Mian Li. Towards accurate and interpretable surgical skill assess- ment: A video-based method incorporating recognized surgical gestures and skill levels. In MICCAI, pages 668- 678, 2020.","title":"Towards accurate and interpretable surgical skill assessment: A video-based method incorporating recognized surgical gestures and skill levels","authors":"Tianyu Wang, Yijie Wang, Mian Li","year":"2020","venue":"MICCAI","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang et al., 2021a] Shunli Wang, Dingkang Yang, Peng Zhai, Chixiao Chen, and Lihua Zhang. Tsa-net: Tube self- attention network for action quality assessment. In ACM MM, pages 4902-4910, 2021.","title":"Tsa-net: Tube selfattention network for action quality assessment","authors":"Wang","year":"2021","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wang et al., 2021b] Shunli Wang, Dingkang Yang, Peng Zhai, Qing Yu, Tao Suo, Zhan Sun, Ka Li, and Lihua Zhang. A survey of video-based action quality assessment. In 2021 International Conference on Networking Systems of AI (INSAI), pages 1-9, 2021.","title":"A survey of video-based action quality assessment","authors":"Wang","year":"2021","venue":"2021 International Conference on Networking Systems of AI (INSAI)","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Xie et al., 2023] Liang Xie, Yibo Yang, Deng Cai, and Xi- aofei He. Neural collapse inspired attraction-repulsion- balanced loss for imbalanced learning. Neurocomputing, 527:60-70, 2023.","title":"Neural collapse inspired attraction-repulsion-balanced loss for imbalanced learning","authors":"Liang Xie, Yibo Yang, Deng Cai, Xiaofei He","year":"2023","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Xu et al., 2019] Chengming Xu, Yanwei Fu, Bing Zhang, Zitian Chen, Yu-Gang Jiang, and Xiangyang Xue. Learn- ing to score figure skating sport videos. IEEE TCSVT, 30(12):4578-4590, 2019.","title":"Learning to score figure skating sport videos","authors":"Chengming Xu, Yanwei Fu, Bing Zhang, Zitian Chen, Yu-Gang Jiang, Xiangyang Xue","year":"2019","venue":"IEEE TCSVT","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Xu et al., 2022] Angchi Xu, Ling-An Zeng, and Wei-Shi Zheng. Likert scoring with grade decoupling for long-term action assessment. In CVPR, pages 3232-3241, 2022.","title":"Likert scoring with grade decoupling for long-term action assessment","authors":"Xu","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Yang et al., 2022] Yibo Yang, Liang Xie, Shixiang Chen, Xiangtai Li, Zhouchen Lin, and Dacheng Tao. Do we re- ally need a learnable classifier at the end of deep neural network? arXiv e-prints, pages arXiv-2203, 2022.","title":"Do we really need a learnable classifier at the end of deep neural network?","authors":"Yibo Yang, Liang Xie, Shixiang Chen, Xiangtai Li, Zhouchen Lin, Dacheng Tao","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Yang et al., 2023] Yibo Yang, Haobo Yuan, Xiangtai Li, Zhouchen Lin, Philip Torr, and Dacheng Tao. Neu- ral collapse inspired feature-classifier alignment for few-shot class incremental learning. arXiv preprint arXiv:2302.03004, 2023.","title":"Neural collapse inspired feature-classifier alignment for few-shot class incremental learning","authors":"Yibo Yang, Haobo Yuan, Xiangtai Li, Zhouchen Lin, Philip Torr, Dacheng Tao","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Yu et al., 2021] Xumin Yu, Yongming Rao, Wenliang Zhao, Jiwen Lu, and Jie Zhou. Group-aware contrastive regres- sion for action quality assessment. In ICCV, pages 7919- 7928, 2021.","title":"Group-aware contrastive regression for action quality assessment","authors":"Xumin Yu, Yongming Rao, Wenliang Zhao, Jiwen Lu, Jie Zhou","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zeng et al., 2020] Ling-An Zeng, Fa-Ting Hong, Wei-Shi Zheng, Qi-Zhi Yu, Wei Zeng, Yao-Wei Wang, and Jian- Huang Lai. Hybrid dynamic-static context-aware attention network for action assessment in long videos. In ACM MM, pages 2526-2534, 2020.","title":"Hybrid dynamic-static context-aware attention network for action assessment in long videos","authors":"Ling-An Zeng, Fa-Ting Hong, Wei-Shi Zheng, Qi-Zhi Yu, Wei Zeng, Yao-Wei Wang, Jian-Huang Lai","year":"2020","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhong et al., 2023] Zhisheng Zhong, Jiequan Cui, Yibo Yang, Xiaoyang Wu, Xiaojuan Qi, Xiangyu Zhang, and Ji- aya Jia. Understanding imbalanced semantic segmentation through neural collapse. In CVPR, pages 19550-19560, 2023.","title":"Understanding imbalanced semantic segmentation through neural collapse","authors":"Zhisheng Zhong, Jiequan Cui, Yibo Yang, Xiaoyang Wu, Xiaojuan Qi, Xiangyu Zhang, Jiaya Jia","year":"2023","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zhou et al., 2022] Jinxin Zhou, Xiao Li, Tianyu Ding, Chong You, Qing Qu, and Zhihui Zhu. On the optimiza- tion landscape of neural collapse under mse loss: Global optimality with unconstrained features. In ICML, pages 27179-27202, 2022.","title":"On the optimization landscape of neural collapse under mse loss: Global optimality with unconstrained features","authors":"Jinxin Zhou, Xiao Li, Tianyu Ding, Chong You, Qing Qu, Zhihui Zhu","year":"2022","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Zhou et al., 2023a] Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. arXiv preprint arXiv:2305.11206, 2023.","title":"Lima: Less is more for alignment","authors":"Zhou","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Zhou et al., 2023b] Kanglei Zhou, Ruizhi Cai, Yue Ma, Qingqing Tan, Xinning Wang, Jianguo Li, Hubert PH Shum, Frederick WB Li, Song Jin, and Xiaohui Liang. A video-based augmented reality system for human-in- the-loop muscle strength assessment of juvenile dermato- myositis. IEEE TVCG, 29(5):2456-2466, 2023.","title":"A video-based augmented reality system for human-inthe-loop muscle strength assessment of juvenile dermatomyositis","authors":"Zhou","year":"2023","venue":"IEEE TVCG","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Zhou et al., 2023c] Kanglei Zhou, Yue Ma, Hubert PH Shum, and Xiaohui Liang. Hierarchical graph convo- lutional networks for action quality assessment. IEEE TCSVT, 2023.","title":"Hierarchical graph convolutional networks for action quality assessment","authors":"Zhou","year":"2023","venue":"IEEE TCSVT","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Zhou et al., 2024] Kanglei Zhou, Liyuan Wang, Xingxing Zhang, Hubert PH Shum, Frederick WB Li, Jianguo Li, and Xiaohui Liang. Magr: Manifold-aligned graph regu- larization for continual action quality assessment. arXiv preprint arXiv:2403.04398, 2024.","title":"Magr: Manifold-aligned graph regularization for continual action quality assessment","authors":"Zhou","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Zhu et al., 2021] Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu. A geometric analysis of neural collapse with unconstrained features. NeurIPS, 34:29820-29834, 2021.","title":"A geometric analysis of neural collapse with unconstrained features","authors":"Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, Qing Qu","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}