{"type":"ScholarlyArticle","title":"Music-to-Dance Generation with Optimal Transport","authors":[{"name":"Shuang Wu","slug":"shuang-wu","given":"Shuang","family":"Wu","affiliation":"Nanyang Technological University"},{"name":"Shijian Lu","slug":"shijian-lu","given":"Shijian","family":"Lu","affiliation":"Nanyang Technological University"},{"name":"Li Cheng","slug":"li-cheng","given":"Li","family":"Cheng","affiliation":"University of Alberta"}],"abstract":"Dance choreography for a piece of music is a challenging task, having to be creative in presenting distinctive stylistic dance elements while taking into account the musical theme and rhythm. It has been tackled by different approaches such as similarity retrieval, sequence-to-sequence modeling and generative adversarial networks, but their generated dance sequences are often short of motion realism, diversity and music consistency. In this paper, we propose a Music-to-Dance with Optimal Transport Network (MDOT-Net) for learning to generate 3D dance choreographies from music. We introduce an optimal transport distance for evaluating the authenticity of the generated dance distribution and a Gromov-Wasserstein distance to measure the correspondence between the dance distribution and the input music. This gives a well defined and non-divergent training objective that mitigates the limitation of standard GAN training which is frequently plagued with instability and divergent generator loss issues. Extensive experiments demonstrate that our MDOT-Net can synthesize realistic and diverse dances which achieve an organic unity with the input music, reflecting the shared intentionality and matching the rhythmic articulation. Sample results are found at https://www.youtube.com/watch?v=dErfBkrlUO8.","doi":"10.24963/ijcai.2022/691","url":"https://ijcai.org/proceedings/2022/691","pdf":"https://ijcai.org/proceedings/2022/0691.pdf","markdown":"https://ijcai.org/proceedings/2022/691.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/691","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Research Papers","track":"AI and Arts","number":691,"pages":{"first":4988,"last":4994},"keywords":["Application domains: Performances, dance","Theory and philosophy of arts and creativity in AI systems: Autonomous creative or artistic AI","Methods and resources: Machine learning, deep learning, neural models, reinforcement learning","Application domains: Music"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987159","length_sec":76,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985302","length_sec":651,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Alvarez-Melis and Jaakkola, 2018] David Alvarez-Melis and Tommi Jaakkola. Gromov-wasserstein alignment of word embedding spaces. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1881-1890, 2018.","title":"Gromov-wasserstein alignment of word embedding spaces","authors":"Alvarez-Melis, David Jaakkola, Tommi Alvarez-Melis, Jaakkola","year":"2018","venue":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Arjovsky et al., 2017] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 214-223, 2017.","title":"Wasserstein generative adversarial networks","authors":"Arjovsky","year":"2017","venue":"Proceedings of the 34th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Artetxe et al., 2018] Mikel Artetxe, Gorka Labaka, and Eneko Agirre. A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 789-798, 2018.","title":"A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings","authors":"Mikel Artetxe, Gorka Labaka, Eneko Agirre","year":"2018","venue":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bunne et al., 2019] Charlotte Bunne, David Alvarez-Melis, An- dreas Krause, and Stefanie Jegelka. Learning generative mod- els across incomparable spaces. In International Conference on Machine Learning, pages 851-861, 2019.","title":"Learning generative models across incomparable spaces","authors":"Bunne","year":"2019","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cuturi, 2013] Marco Cuturi. Sinkhorn distances: Lightspeed com- putation of optimal transport. In Advances in neural information processing systems, pages 2292-2300, 2013.","title":"Sinkhorn distances: Lightspeed computation of optimal transport","authors":"Marco Cuturi","year":"2013","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fan et al., 2011] Rukun Fan, Songhua Xu, and Weidong Geng. Example-based automatic music-driven conventional dance mo- tion synthesis. IEEE transactions on visualization and computer graphics, 18(3):501-515, 2011.","title":"Example-based automatic music-driven conventional dance motion synthesis","authors":"Rukun Fan, Songhua Xu, Weidong Geng","year":"2011","venue":"IEEE transactions on visualization and computer graphics","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Flamary et al., 2021] Rémi Flamary, Nicolas Courty, Alexandre Gramfort, Mokhtar Z. Alaya, Aurélie Boisbunon, Stanislas Chambon, Laetitia Chapel, Adrien Corenflos, Kilian Fatras, Nemo Fournier, Léo Gautheron, Nathalie T.H. Gayraud, Hicham Janati, Alain Rakotomamonjy, Ievgen Redko, Antoine Rolet, Antony Schutz, Vivien Seguy, Danica J. Sutherland, Romain Tavenard, Alexander Tong, and Titouan Vayer. Pot: Python opti- mal transport. Journal of Machine Learning Research, 22(78):1- 8, 2021.","title":"Pot: Python optimal transport","authors":"Rémi Flamary, Nicolas Courty, Alexandre Gramfort, Z Mokhtar, Aurélie Alaya, Stanislas Boisbunon, Laetitia Chambon, Adrien Chapel, Kilian Corenflos, Nemo Fatras, Léo Fournier, Nathalie T H Gautheron, Hicham Gayraud, Alain Janati, Ievgen Rakotomamonjy, Antoine Redko, Antony Rolet, Vivien Schutz, Danica J Seguy, Romain Sutherland, Alexander Tavenard, Titouan Tong, Vayer","year":"2021","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Genevay et al., 2016] Aude Genevay, Marco Cuturi, Gabriel Peyré, and Francis Bach. Stochastic optimization for large-scale optimal transport. In Advances in neural information processing systems, pages 3440-3448, 2016.","title":"Stochastic optimization for large-scale optimal transport","authors":"Aude Genevay, Marco Cuturi, Gabriel Peyré, Francis Bach","year":"2016","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Genevay et al., 2018] Aude Genevay, Gabriel Peyré, and Marco Cuturi. Learning generative models with sinkhorn divergences. In International Conference on Artificial Intelligence and Statis- tics, pages 1608-1617, 2018.","title":"Learning generative models with sinkhorn divergences","authors":"Aude Genevay, Gabriel Peyré, Marco Cuturi","year":"2018","venue":"International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Heusel et al., 2017] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in neural information processing sys- tems, pages 6626-6637, 2017.","title":"Gans trained by a two time-scale update rule converge to a local nash equilibrium","authors":"Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, Sepp Hochreiter","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Huang et al., 2021] Ruozi Huang, Huang Hu, Wei Wu, Kei Sawada, Mi Zhang, and Daxin Jiang. Dance revolution: Long- term dance generation with music via curriculum learning. In International Conference on Learning Representations, 2021.","title":"Dance revolution: Longterm dance generation with music via curriculum learning","authors":"Ruozi Huang, Huang Hu, Wei Wu, Kei Sawada, Mi Zhang, Daxin Jiang","year":"2021","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kantorovitch, 1958] Leonid Kantorovitch. On the translocation of masses. Management science, 5(1):1-4, 1958.","title":"Leonid Kantorovitch. On the translocation of masses","authors":"Kantorovitch","year":"1958","venue":"Management science","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lee et al., 2013] Minho Lee, Kyogu Lee, and Jaeheung Park. Mu- sic similarity-based approach to generating dance motion se- quence. Multimedia tools and applications, 62(3):895-912, 2013.","title":"Minho Lee, Kyogu Lee, and Jaeheung Park. Music similarity-based approach to generating dance motion sequence","authors":"Lee","year":"2013","venue":"Multimedia tools and applications","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lee et al., 2019] Hsin-Ying Lee, Xiaodong Yang, Ming-Yu Liu, Ting-Chun Wang, Yu-Ding Lu, Ming-Hsuan Yang, and Jan Kautz. Dancing to music. In Advances in Neural Information Processing Systems, pages 3586-3596, 2019.","title":"Dancing to music","authors":"Hsin-Ying Lee, Xiaodong Yang, Ming-Yu Liu, Ting-Chun Wang, Yu-Ding Lu, Ming-Hsuan Yang, Jan Kautz","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2021] Ruilong Li, Shan Yang, David A Ross, and Angjoo Kanazawa. Ai choreographer: Music conditioned 3d dance gen- eration with aist++. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 13401-13412, 2021.","title":"Ai choreographer: Music conditioned 3d dance generation with aist++","authors":"Ruilong Li, Shan Yang, David A Ross, Angjoo Kanazawa","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Loper et al., 2015] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. SMPL: A skinned multi-person linear model. ACM Trans. Graphics (Proc. SIGGRAPH Asia), 34(6):248:1-248:16, October 2015.","title":"SMPL: A skinned multi-person linear model","authors":"Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, Michael J Black","year":"2015","venue":"ACM Trans. Graphics (Proc. SIGGRAPH Asia)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[McFee et al., 2015] Brian McFee, Colin Raffel, Dawen Liang, Daniel PW Ellis, Matt McVicar, Eric Battenberg, and Oriol Ni- eto. librosa: Audio and music signal analysis in python. In Proceedings of the 14th python in science conference, volume 8, pages 18-25, 2015.","title":"librosa: Audio and music signal analysis in python","authors":"Brian Mcfee, Colin Raffel, Dawen Liang, P W Daniel, Matt Ellis, Eric Mcvicar, Oriol Battenberg, Nieto","year":"2015","venue":"Proceedings of the 14th python in science conference","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mémoli, 2011] Facundo Mémoli. Gromov-wasserstein distances and the metric approach to object matching. Foundations of com- putational mathematics, 11(4):417-487, 2011.","title":"Gromov-wasserstein distances and the metric approach to object matching","authors":"Facundo Mémoli","year":"2011","venue":"Foundations of computational mathematics","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Müller, 2015] Meinard Müller. Fundamentals of music process- ing: Audio, analysis, algorithms, applications. Springer, 2015.","title":"Meinard Müller. Fundamentals of music processing: Audio, analysis, algorithms, applications","authors":"Müller","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Ren et al., 2020] Xuanchi Ren, Haoran Li, Zijian Huang, and Qifeng Chen. Self-supervised dance video synthesis conditioned on music. In Proceedings of the 28th ACM International Confer- ence on Multimedia, pages 46-54, 2020.","title":"Self-supervised dance video synthesis conditioned on music","authors":"Ren","year":"2020","venue":"Proceedings of the 28th ACM International Conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Salimans et al., 2018] Tim Salimans, Han Zhang, Alec Radford, and Dimitris Metaxas. Improving gans using optimal transport. In International Conference on Learning Representations, 2018.","title":"Improving gans using optimal transport","authors":"Tim Salimans, Han Zhang, Alec Radford, Dimitris Metaxas","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Shi et al., 2019] Lei Shi, Yifan Zhang, Jian Cheng, and Hanqing Lu. Two-stream adaptive graph convolutional networks for skeleton-based action recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 12026-12035, 2019.","title":"Two-stream adaptive graph convolutional networks for skeleton-based action recognition","authors":"Lei Shi, Yifan Zhang, Jian Cheng, Hanqing Lu","year":"2019","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Tang et al., 2018] Taoran Tang, Jia Jia, and Hanyang Mao. Dance with melody: An lstm-autoencoder approach to music-oriented dance synthesis. In Proceedings of the 26th ACM international conference on Multimedia, pages 1598-1606, 2018.","title":"Dance with melody: An lstm-autoencoder approach to music-oriented dance synthesis","authors":"Taoran Tang, Jia Jia, Hanyang Mao","year":"2018","venue":"Proceedings of the 26th ACM international conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Valle-Pérez et al., 2021] Guillermo Valle-Pérez, Gustav Eje Hen- ter, Jonas Beskow, Andre Holzapfel, Pierre-Yves Oudeyer, and Simon Alexanderson. Transflower: probabilistic autoregressive dance generation with multimodal attention. ACM Transactions on Graphics (TOG), 40(6):1-14, 2021.","title":"Transflower: probabilistic autoregressive dance generation with multimodal attention","authors":"Valle-Pérez","year":"2021","venue":"ACM Transactions on Graphics (TOG)","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Villani, 2008] Cédric Villani. Optimal transport: old and new, vol- ume 338. Springer Science & Business Media, 2008.","title":"Optimal transport: old and new","authors":"Villani","year":"2008","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wu et al., 2021] Shuang Wu, Zhenguang Liu, Shijian Lu, and Li Cheng. Dual learning music composition and dance choreog- raphy. In Proceedings of the 29th ACM International Conference on Multimedia, pages 3746-3754, 2021.","title":"Dual learning music composition and dance choreography","authors":"Shuang Wu, Zhenguang Liu, Shijian Lu, Li Cheng","year":"2021","venue":"Proceedings of the 29th ACM International Conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Xu et al., 2019] Hongteng Xu, Dixin Luo, Hongyuan Zha, and Lawrence Carin Duke. Gromov-wasserstein learning for graph matching and node embedding. In International conference on machine learning, pages 6932-6941. PMLR, 2019.","title":"Gromov-wasserstein learning for graph matching and node embedding","authors":"Hongteng Xu, Dixin Luo, Hongyuan Zha, Lawrence Carin, Duke","year":"2019","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhang et al., 2018] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effective- ness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018.","title":"The unreasonable effectiveness of deep features as a perceptual metric","authors":"Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, Oliver Wang","year":"2018","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}