{"type":"ScholarlyArticle","title":"M2Beats: When Motion Meets Beats in Short-form Videos","authors":[{"name":"Dongxiang Jiang","slug":"dongxiang-jiang","given":"Dongxiang","family":"Jiang","affiliation":"School of Computer Science (National Pilot Software Engineering School) , Beijing University of Posts and Telecommunications"},{"name":"Yongchang Zhang","slug":"yongchang-zhang","given":"Yongchang","family":"Zhang","affiliation":"School of Computer Science (National Pilot Software Engineering School) , Beijing University of Posts and Telecommunications"},{"name":"Shuai He","slug":"shuai-he","given":"Shuai","family":"He","affiliation":"School of Computer Science (National Pilot Software Engineering School) , Beijing University of Posts and Telecommunications"},{"name":"Anlong Ming","slug":"anlong-ming","given":"Anlong","family":"Ming","affiliation":"School of Computer Science (National Pilot Software Engineering School) , Beijing University of Posts and Telecommunications"}],"abstract":"In recent years, short-form videos have gained popularity and the editing of these videos, particularly when motion is synchronized with music,  is highly favored due to its beat-matching effect.  However, detecting motion rhythm poses a significant challenge as it is influenced by multiple factors that make it difficult to define using explicit  rules.  While traditional methods attempt to define  motion rhythm, they often yield unsatisfactory results.  On the other hand, learning-based methods  can extract motion rhythm without relying on explicit rules but require high-quality datasets.  Unfortunately, existing datasets simply substitute music rhythm for motion rhythm which are not equivalent.  To address these challenges, we present the  motion rhythm dataset AIST-M2B, which is annotated with meticulously curated motion rhythm labels derived from the profound correlation between  motion and music in professional dance.  We propose a novel network architecture called M2BNet  that is specifically trained on AIST-M2B to effectively extract intricate motion rhythms by incorporating both human body structure and temporal information.  Additionally, we introduce a pioneering  algorithm for enhancing motion rhythm synchronization with beats.  Experimental results substan-  tiate the superior performance of our method compared to other existing algorithms in the domain of  motion rhythm analysis.  Our code is available at  https://github.com/mRobotit/M2Beats.","doi":"10.24963/ijcai.2024/102","url":"https://ijcai.org/proceedings/2024/102","pdf":"https://ijcai.org/proceedings/2024/0102.pdf","markdown":"https://ijcai.org/proceedings/2024/102.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/102","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Computer Vision","track":"Main Track","number":102,"pages":{"first":920,"last":928},"keywords":["Computer Vision: CV: Image and video synthesis and generation","Computer Vision: CV: Motion and tracking","Computer Vision: CV: Image and video retrieval"],"videos":[],"references":[{"position":1,"raw":"[Aasen, 2021] Solveig Aasen. Crossmodal aesthetics: How music and dance can match. The Philosophical Quarterly, 71(2):223-240, 2021.","title":"Crossmodal aesthetics: How music and dance can match","authors":"Solveig Aasen, Aasen","year":"2021","venue":"The Philosophical Quarterly","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bellini et al., 2018] Rachele Bellini, Yanir Kleiman, and Daniel Cohen-Or. Dance to the beat: Synchronizing mo- tion to audio. Computational Visual Media, 4:197-208, 2018.","title":"Dance to the beat: Synchronizing motion to audio","authors":"Rachele Bellini, Yanir Kleiman, Daniel Cohen-Or","year":"2018","venue":"Computational Visual Media","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Böck and Widmer, 2013] Sebastian Böck and Gerhard Wid- mer. Maximum filter vibrato suppression for onset detec- tion. In Proc. of the 16th Int. Conf. on Digital Audio Effects (DAFx). Maynooth, Ireland (Sept 2013), volume 7, page 4, 2013.","title":"Maximum filter vibrato suppression for onset detection","authors":"Widmer Böck, Sebastian Böck, Gerhard Widmer","year":"2013","venue":"Proc. of the 16th Int. Conf. on Digital Audio Effects (DAFx)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Böck et al., 2016] Sebastian Böck, Filip Korzeniowski, Jan Schlüter, Florian Krebs, and Gerhard Widmer. Madmom: A new python audio and music signal processing library. In Proceedings of the 24th ACM international conference on Multimedia, pages 1174-1178, 2016.","title":"Madmom: A new python audio and music signal processing library","authors":"Sebastian Böck, Filip Korzeniowski, Jan Schlüter, Florian Krebs, Gerhard Widmer","year":"2016","venue":"Proceedings of the 24th ACM international conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bredin and Chollet, 2007] Hervé Bredin and Gérard Chol- let. Audiovisual speech synchrony measure: application to biometrics. EURASIP Journal on Advances in Signal Processing, 2007:1-11, 2007.","title":"Audiovisual speech synchrony measure: application to biometrics","authors":"Chollet Bredin, Hervé Bredin, Gérard Chollet","year":"2007","venue":"EURASIP Journal on Advances in Signal Processing","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen et al., 2021] Kang Chen, Zhipeng Tan, Jin Lei, Song- Hai Zhang, Yuan-Chen Guo, Weidong Zhang, and Shi- Min Hu. Choreomaster: choreography-oriented music- driven dance synthesis. ACM Transactions on Graphics (TOG), 40(4):1-13, 2021.","title":"Choreomaster: choreography-oriented musicdriven dance synthesis","authors":"Kang Chen, Zhipeng Tan, Jin Lei, Song-Hai Zhang, Yuan-Chen Guo, Weidong Zhang, Shi-Min Hu","year":"2021","venue":"ACM Transactions on Graphics (TOG)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chu and Tsai, 2011] Wei-Ta Chu and Shang-Yin Tsai. Rhythm of motion extraction and rhythm-based cross- media alignment for dance videos. IEEE Transactions on Multimedia, 14(1):129-141, 2011.","title":"Rhythm of motion extraction and rhythm-based crossmedia alignment for dance videos","authors":"Chu, Tsai ; ] Wei-Ta Chu, Shang-Yin Tsai","year":"2011","venue":"IEEE Transactions on Multimedia","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chung and Zisserman, 2017] Joon Son Chung and Andrew Zisserman. Out of time: automated lip sync in the wild. In Computer Vision-ACCV 2016 Workshops: ACCV 2016 International Workshops, Taipei, Taiwan, November 20- 24, 2016, Revised Selected Papers, Part II 13, pages 251- 263. Springer, 2017.","title":"Out of time: automated lip sync in the wild","authors":"Chung, Zisserman ; ] Joon, Son Chung, Andrew Zisserman","year":"2016","venue":"Computer Vision-ACCV 2016 Workshops: ACCV 2016 International Workshops","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Davis and Agrawala, 2018] Abe Davis and Maneesh Agrawala. Visual rhythm and beat. ACM Transactions on Graphics (TOG), 37(4):1-11, 2018.","title":"Abe Davis and Maneesh Agrawala. Visual rhythm and beat","authors":"Agrawala Davis","year":"2018","venue":"ACM Transactions on Graphics (TOG)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ge et al., 2021] Zheng Ge, Songtao Liu, Feng Wang, Zem- ing Li, and Jian Sun. Yolox: Exceeding yolo series in 2021. arXiv preprint arXiv:2107.08430, 2021.","title":"Yolox: Exceeding yolo series in 2021","authors":"Zheng Ge, Songtao Liu, Feng Wang, Zeming Li, Jian Sun","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Halperin et al., 2019] Tavi Halperin, Ariel Ephrat, and Shmuel Peleg. Dynamic temporal alignment of speech to lips. In ICASSP 2019-2019 IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP), pages 3980-3984. IEEE, 2019.","title":"Dynamic temporal alignment of speech to lips","authors":"Tavi Halperin, Ariel Ephrat, Shmuel Peleg","year":"2019","venue":"ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[He et al., 2022] Shuai He, Yongchang Zhang, Rui Xie, Dongxiang Jiang, and Anlong Ming. Rethinking image aesthetics assessment: Models, datasets and benchmarks. In IJCAI, pages 942-948, 2022.","title":"Rethinking image aesthetics assessment: Models, datasets and benchmarks","authors":"Shuai He, Yongchang Zhang, Rui Xie, Dongxiang Jiang, Anlong Ming","year":"2022","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2022/132"},{"position":14,"raw":"[He et al., 2023] Shuai He, Anlong Ming, Shuntian Zheng, Haobin Zhong, and Huadong Ma. Eat: An enhancer for aesthetics-oriented transformers. In Proceedings of the 31st ACM International Conference on Multimedia, pages 1023-1032, 2023.","title":"Eat: An enhancer for aesthetics-oriented transformers","authors":"Shuai He, Anlong Ming, Shuntian Zheng, Haobin Zhong, Huadong Ma","year":"2023","venue":"Proceedings of the 31st ACM International Conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kim et al., 2003] Tae-hoon Kim, Sang Il Park, and Sung Yong Shin. Rhythmic-motion synthesis based on motion-beat analysis. ACM Transactions on Graphics (TOG), 22(3):392-401, 2003.","title":"Rhythmic-motion synthesis based on motion-beat analysis","authors":"Kim","year":"2003","venue":"ACM Transactions on Graphics (TOG)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liu et al., 2020] Ziyu Liu, Hongwen Zhang, Zhenghao Chen, Zhiyong Wang, and Wanli Ouyang. Disentangling and unifying graph convolutions for skeleton-based action recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 143- 152, 2020.","title":"Disentangling and unifying graph convolutions for skeleton-based action recognition","authors":"Ziyu Liu, Hongwen Zhang, Zhenghao Chen, Zhiyong Wang, Wanli Ouyang","year":"2020","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[McFee et al., 2015] Brian McFee, Colin Raffel, Dawen Liang, Daniel P Ellis, Matt McVicar, Eric Battenberg, and Oriol Nieto. librosa: Audio and music signal analysis in python. In Proceedings of the 14th python in science con- ference, volume 8, pages 18-25, 2015.","title":"librosa: Audio and music signal analysis in python","authors":"Brian Mcfee, Colin Raffel, Dawen Liang, P Daniel, Matt Ellis, Eric Mcvicar, Oriol Battenberg, Nieto","year":"2015","venue":"Proceedings of the 14th python in science conference","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Owens and Efros, 2018] Andrew Owens and Alexei A Efros. Audio-visual scene analysis with self-supervised multisensory features. In Proceedings of the European conference on computer vision (ECCV), pages 631-648, 2018.","title":"Audio-visual scene analysis with self-supervised multisensory features","authors":"Andrew Owens, Alexei A Efros","year":"2018","venue":"Proceedings of the European conference on computer vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sargin et al., 2007] Mehmet Emre Sargin, Yücel Yemez, Engin Erzin, and A Murat Tekalp. Audiovisual synchro- nization and fusion using canonical correlation analysis. IEEE transactions on Multimedia, 9(7):1396-1403, 2007.","title":"Audiovisual synchronization and fusion using canonical correlation analysis","authors":"Yücel Mehmet Emre Sargin, Engin Yemez, Erzin, Tekalp Murat","year":"2007","venue":"IEEE transactions on Multimedia","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Shiratori and Ikeuchi, 2008] Takaaki Shiratori and Katsushi Ikeuchi. Synthesis of dance performance based on analy- ses of human motion and music. Information and Media Technologies, 3(4):834-847, 2008.","title":"Synthesis of dance performance based on analyses of human motion and music","authors":"Ikeuchi Shiratori, Takaaki Shiratori, Katsushi Ikeuchi","year":"2008","venue":"Information and Media Technologies","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Shiratori et al., 2004] Takaaki Shiratori, Atsushi Nakazawa, and Katsushi Ikeuchi. Detecting dance motion structure through music analysis. In Sixth IEEE International Con- ference on Automatic Face and Gesture Recognition, 2004. Proceedings., pages 857-862. IEEE, 2004.","title":"Detecting dance motion structure through music analysis","authors":"Takaaki Shiratori, Atsushi Nakazawa, Katsushi Ikeuchi","year":"2004","venue":"Sixth IEEE International Conference on Automatic Face and Gesture Recognition","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Shiratori et al., 2006a] Takaaki Shiratori, Atsushi Nakazawa, and Katsushi Ikeuchi. Dancing-to-music character animation. In Computer Graphics Forum, volume 25, pages 449-458. Wiley Online Library, 2006.","title":"Dancing-to-music character animation","authors":"Shiratori","year":"2006","venue":"Computer Graphics Forum","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Shiratori et al., 2006b] Takaaki Shiratori, Atsushi Nakazawa, and Katsushi Ikeuchi. Dancing-to-music character animation. In Computer Graphics Forum, volume 25, pages 449-458. Wiley Online Library, 2006.","title":"Dancing-to-music character animation","authors":"Shiratori","year":"2006","venue":"Computer Graphics Forum","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Song et al., 2020] Chao Song, Youfang Lin, Shengnan Guo, and Huaiyu Wan. Spatial-temporal synchronous graph convolutional networks: A new framework for spatial- temporal network data forecasting. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 914-921, 2020.","title":"Spatial-temporal synchronous graph convolutional networks: A new framework for spatialtemporal network data forecasting","authors":"Chao Song, Youfang Lin, Shengnan Guo, Huaiyu Wan","year":"2020","venue":"Proceedings of the AAAI conference on artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Tchemeube et al., 2023] Renaud Bougueng Tchemeube, Jeffrey Ens, Cale Plut, Philippe Pasquier, Maryam Safi, Yvan Grabit, and Jean-Baptiste Rolland. Evaluating human-ai interaction via usability, user experience and ac- ceptance measures for mmm-c: A creative ai system for music composition. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, pages 5769-5778, 2023.","title":"Evaluating human-ai interaction via usability, user experience and acceptance measures for mmm-c: A creative ai system for music composition","authors":"Renaud Bougueng Tchemeube, Jeffrey Ens, Cale Plut, Philippe Pasquier, Maryam Safi, Yvan Grabit, Jean-Baptiste Rolland","year":"2023","venue":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2023/640"},{"position":26,"raw":"[Wang et al., 2020] Jianren Wang, Zhaoyuan Fang, and Hang Zhao. Alignnet: A unifying approach to audio-visual alignment. In Proceedings of the IEEE/CVF Winter Con- ference on Applications of Computer Vision, pages 3309- 3317, 2020.","title":"Alignnet: A unifying approach to audio-visual alignment","authors":"Wang","year":"2020","venue":"Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yu et al., 2017] Bing Yu, Haoteng Yin, and Zhanxing Zhu. Spatio-temporal graph convolutional networks: A deep learning framework for traffic forecasting. arXiv preprint arXiv:1709.04875, 2017.","title":"Spatio-temporal graph convolutional networks: A deep learning framework for traffic forecasting","authors":"Yu","year":"2017","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2018/505"},{"position":28,"raw":"[Yu et al., 2023] Jiashuo Yu, Junfu Pu, Ying Cheng, Rui Feng, and Ying Shan. Learning music-dance represen- tations through explicit-implicit rhythm synchronization. IEEE Transactions on Multimedia, pages 1-10, 2023.","title":"Learning music-dance representations through explicit-implicit rhythm synchronization","authors":"Yu","year":"2023","venue":"IEEE Transactions on Multimedia","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhou et al., 2023] Qiu Zhou, Manyi Li, Qiong Zeng, An- dreas Aristidou, Xiaojing Zhang, Lin Chen, and Changhe Tu. Let's all dance: Enhancing amateur dance motions. Computational Visual Media, 9(3):531-550, 2023.","title":"Let's all dance: Enhancing amateur dance motions","authors":"Qiu Zhou, Manyi Li, Qiong Zeng, Andreas Aristidou, Xiaojing Zhang, Lin Chen, Changhe Tu","year":"2023","venue":"Computational Visual Media","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}