{"type":"ScholarlyArticle","title":"Musical Composition Style Transfer via Disentangled Timbre Representations","authors":[{"name":"Yun-Ning Hung","slug":"yun-ning-hung","given":"Yun-Ning","family":"Hung","affiliation":"Research Center for IT Innovation , Academia Sinica , Taiwan"},{"name":"I-Tung Chiang","slug":"i-tung-chiang","given":"I-Tung","family":"Chiang","affiliation":"Research Center for IT Innovation , Academia Sinica , Taiwan"},{"name":"Yi-An Chen","slug":"yi-an-chen","given":"Yi-An","family":"Chen","affiliation":"KKBOX Inc. , Taiwan"},{"name":"Yi-Hsuan Yang","slug":"yi-hsuan-yang","given":"Yi-Hsuan","family":"Yang","affiliation":"Research Center for IT Innovation , Academia Sinica , Taiwan"}],"abstract":"Music creation involves not only composing the different parts (e.g., melody, chords) of a musical work but also arranging/selecting the instruments to play the different parts. While the former has received increasing attention, the latter has not been much investigated. This paper presents, to the best\r\nof our knowledge, the first deep learning models for rearranging music of arbitrary genres. Specifically, we build encoders and decoders that take a\r\npiece of polyphonic musical audio as input, and predict as output its musical score. We investigate disentanglement techniques such as adversarial\r\ntraining to separate latent factors that are related to the musical content (pitch) of different parts of the piece, and that are related to the instrumentation\r\n(timbre) of the parts per short-time segment. By disentangling pitch and timbre, our models have an idea of how each piece was composed and arranged. Moreover, the models can realize “composition style transfer” by rearranging a musical piece without much affecting its pitch content. We\r\nvalidate the effectiveness of the models by experiments on instrument activity detection and composition style transfer. To facilitate follow-up research,\r\nwe open source our code at https://github.com/biboamy/instrument-disentangle.","doi":"10.24963/ijcai.2019/652","url":"https://ijcai.org/proceedings/2019/652","pdf":"https://ijcai.org/proceedings/2019/0652.pdf","markdown":"https://ijcai.org/proceedings/2019/652.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/652","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Multidisciplinary Topics and Applications","track":"Main track","number":652,"pages":{"first":4697,"last":4703},"keywords":["Multidisciplinary Topics and Applications: Information Retrieval","Machine Learning: Transfer, Adaptation, Multi-task Learning","Multidisciplinary Topics and Applications: Art and Music","Machine Learning: Adversarial Machine Learning","Machine Learning: Deep Learning"],"videos":[],"references":[{"position":1,"raw":"[Bittner et al., 2017] Rachel M. Bittner, Brian McFee, Justin Salamon, Peter Li, and Juan P. Bello. Deep salience rep- resentations for f 0 tracking in polyphonic music. In Proc. Int. Soc. Music Information Retrieval Conf., 2017.","title":"Deep salience representations for f 0 tracking in polyphonic music","authors":"Rachel M Bittner, Brian Mcfee, Justin Salamon, Peter Li, Juan P Bello","year":"2017","venue":"Proc. Int. Soc. Music Information Retrieval Conf","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Briot et al., 2017] Jean-Pierre Briot, Gaëtan Hadjeres, and Franc ¸ois Pachet. Deep learning techniques for music gen- eration: A survey. arXiv preprint arXiv:1709.01620, 2017.","title":"Deep learning techniques for music generation: A survey","authors":"Jean-Pierre Briot, Gaëtan Hadjeres, Franc ¸ois, Pachet","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Brunner et al., 2018] Gino Brunner, Andres Konrad, Yuyi Wang, and Roger Wattenhofer. MIDI-VAE: Modeling dy- namics and instrumentation of music with applications to style transfer. arXiv preprint arXiv:1809.07600, 2018.","title":"MIDI-VAE: Modeling dynamics and instrumentation of music with applications to style transfer","authors":"Gino Brunner, Andres Konrad, Yuyi Wang, Roger Wattenhofer","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Corozine, 2015] Vince Corozine. Arranging Music for the Real World. Mel Bay Publications, 2015.","title":"Arranging Music for the Real World","authors":"Vince Corozine","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Crestel et al., 2017] Léopold Crestel, Philippe Esling, Lena Heng, and Stephen McAdams. A database linking piano and orchestral MIDI scores with application to automatic projective orchestration. In Proc. Int. Soc. Music Informa- tion Retrieval Conf., pages 592-598, 2017.","title":"A database linking piano and orchestral MIDI scores with application to automatic projective orchestration","authors":"Léopold Crestel, Philippe Esling, Lena Heng, Stephen Mcadams","year":"2017","venue":"Proc. Int. Soc. Music Information Retrieval Conf","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dai et al., 2018] Shuqi Dai, Zheng Zhang, and Gus Xia. Music style transfer issues: A position paper. arXiv preprint arXiv:1803.06841, 2018.","title":"Music style transfer issues: A position paper","authors":"Dai","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dong et al., 2018a] Hao-Wen Dong, Wen-Yi Hsiao, Li- Chia Yang, and Yi-Hsuan Yang. MuseGAN: Multi-track sequential generative adversarial networks for symbolic music generation and accompaniment. In Proc. AAAI Conf. Artificial Intelligence, 2018.","title":"MuseGAN: Multi-track sequential generative adversarial networks for symbolic music generation and accompaniment","authors":"Dong","year":"2018","venue":"Proc. AAAI Conf. Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dong et al., 2018b] Hao-Wen Dong, Wen-Yi Hsiao, and Yi- Hsuan Yang. Pypianoroll: Open source Python package for handling multitrack pianoroll. In Proc. Int. Soc. Music Information Retrieval Conf., 2018. Late-breaking paper.","title":"Pypianoroll: Open source Python package for handling multitrack pianoroll","authors":"Dong","year":"2018","venue":"Proc. Int. Soc. Music Information Retrieval Conf","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Engel et al., 2017] Jesse Engel, Cinjon Resnick, Adam Roberts, Sander Dieleman, Mohammad Norouzi, Douglas Eck, and Karen Simonyan. Neural audio synthesis of mu- sical notes with wavenet autoencoders. In Proc. Int. Conf. Machine Learning, pages 1068-1077, 2017.","title":"Neural audio synthesis of musical notes with wavenet autoencoders","authors":"Jesse Engel, Cinjon Resnick, Adam Roberts, Sander Dieleman, Mohammad Norouzi, Douglas Eck, Karen Simonyan","year":"2017","venue":"Proc. Int. Conf. Machine Learning","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gatys et al., 2016] Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In Proc. IEEE Conf. Computer Vision and Pattern Recognition, pages 2414-2423, 2016.","title":"Image style transfer using convolutional neural networks","authors":"Leon A Gatys, Alexander S Ecker, Matthias Bethge","year":"2016","venue":"Proc. IEEE Conf. Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Goodfellow and others, 2014] Ian Goodfellow et al. Gener- ative adversarial nets. In Advances in neural information processing systems, pages 2672-2680, 2014.","title":"Generative adversarial nets","authors":"Ian Goodfellow","year":"2014","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Gururani and Lerch, 2017] Siddharth Gururani and Alexan- der Lerch. Mixing Secrets: A multi-track dataset for instrument recognition in polyphonic music. In Proc. Int. Soc. Music Information Retrieval Conf., 2017. Late- breaking paper.","title":"Siddharth Gururani and Alexander Lerch. Mixing Secrets: A multi-track dataset for instrument recognition in polyphonic music","authors":"Lerch Gururani","year":"2017","venue":"Proc. Int. Soc. Music Information Retrieval Conf","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Gururani et al., 2018] Siddharth Gururani, Cameron Sum- mers, and Alexander Lerch. Instrument activity detection in polyphonic music using deep neural networks. In Proc. Int. Soc. Music Information Retrieval Conf., 2018.","title":"Instrument activity detection in polyphonic music using deep neural networks","authors":"Siddharth Gururani, Cameron Summers, Alexander Lerch","year":"2018","venue":"Proc. Int. Soc. Music Information Retrieval Conf","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hadad et al., 2018] Naama Hadad, Lior Wolf, and Moni Shahar. A two-step disentanglement method. In Proc. IEEE Conf. Computer Vision and Pattern Recognition, pages 772-780, 2018.","title":"A two-step disentanglement method","authors":"Naama Hadad, Lior Wolf, Moni Shahar","year":"2018","venue":"Proc. IEEE Conf. Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Hung and Yang, 2018] Yun-Ning Hung and Yi-Hsuan Yang. Frame-level instrument recognition by timbre and pitch. In Proc. Int. Soc. Music Information Retrieval Conf., pages 135-142, 2018.","title":"Frame-level instrument recognition by timbre and pitch","authors":"Yun-Ning Hung, Yi-Hsuan Yang","year":"2018","venue":"Proc. Int. Soc. Music Information Retrieval Conf","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Hung et al., 2018] Yun-Ning Hung, Yi-An Chen, and Yi- Hsuan Yang. Learning disentangled representations for timber and pitch in music audio. arXiv preprint arXiv:1811.03271, 2018.","title":"Learning disentangled representations for timber and pitch in music audio","authors":"Yun-Ning Hung, Yi-An Chen, Yi-Hsuan Yang","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Hung et al., 2019] Yun-Ning Hung, Yi-An Chen, and Yi- Hsuan Yang. Multitask learning for frame-level instrument recognition. In Proc. IEEE Int. Conf. Acoustics, Speech and Signal Processing, 2019.","title":"Multitask learning for frame-level instrument recognition","authors":"Yun-Ning Hung, Yi-An Chen, Yi-Hsuan Yang","year":"2019","venue":"Proc. IEEE Int. Conf. Acoustics, Speech and Signal Processing","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kaliakatsos-Papakostas et al., 2017] Maximos Kaliakatsos- Papakostas, Marcelo Queiroz, Costas Tsougras, and Emil- ios Cambouropoulos. Conceptual blending of harmonic spaces for creative melodic harmonisation. J. New Music Research, 46(4):305-328, 2017.","title":"Conceptual blending of harmonic spaces for creative melodic harmonisation","authors":"Maximos Kaliakatsos-Papakostas, Marcelo Queiroz, Costas Tsougras","year":"2017","venue":"J. New Music Research","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lee et al., 2018] Hsin-Ying Lee, Hung-Yu Tseng, Jia-Bin Huang, Maneesh Singh, and Ming-Hsuan Yang. Diverse image-to-image translation via disentangled representa- tions. In Proc. European Conf. Computer Vision, 2018.","title":"Diverse image-to-image translation via disentangled representations","authors":"Hsin-Ying Lee, Hung-Yu Tseng, Jia-Bin Huang, Maneesh Singh, Ming-Hsuan Yang","year":"2018","venue":"Proc. European Conf. Computer Vision","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Liu et al., 2018a] Yang Liu, Zhaowen Wang, Hailin Jin, and Ian Wassell. Multi-task adversarial network for disentan- gled feature learning. In Proc. IEEE Conf. Computer Vi- sion and Pattern Recognition, pages 3743-3751, 2018.","title":"Multi-task adversarial network for disentangled feature learning","authors":"Liu","year":"2018","venue":"Proc. IEEE Conf. Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Liu et al., 2018b] Yu Liu, Fangyin Wei, Jing Shao, Lu Sheng, Junjie Yan, and Xiaogang Wang. Exploring disentangled feature representation beyond face identifi- cation. In Proc. IEEE Conf. Computer Vision and Pattern Recognition, pages 2080-2089, 2018.","title":"Exploring disentangled feature representation beyond face identification","authors":"Liu","year":"2018","venue":"Proc. IEEE Conf. Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Liu et al., 2019] Jen-Yu Liu, Yi-Hsuan Yang, and Shyh- Kang Jeng. Weakly-supervised visual instrument-playing action detection in videos. IEEE Trans. Multimedia, 2019.","title":"Weakly-supervised visual instrument-playing action detection in videos","authors":"Jen-Yu Liu, Yi-Hsuan Yang, Shyh-Kang Jeng","year":"2019","venue":"IEEE Trans. Multimedia","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Lu et al., 2019] Chien-Yu Lu, Min-Xin Xue, Chia-Che Chang, Che-Rung Lee, and Li Su. Play as you like: Timbre-enhanced multi-modal music style transfer. In Proc. AAAI Conf. Artificial Intelligence, 2019.","title":"Play as you like: Timbre-enhanced multi-modal music style transfer","authors":"Chien-Yu Lu, Min-Xin Xue, Chia-Che Chang, Che-Rung Lee, Li Su","year":"2019","venue":"Proc. AAAI Conf. Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[McFee et al., 2015] Brian McFee, Colin Raffel, Dawen Liang, Daniel PW Ellis, Matt McVicar, Eric Battenberg, and Oriol Nieto. librosa: Audio and music signal analysis in python. In Proc. Python in Science Conf., 2015.","title":"librosa: Audio and music signal analysis in python","authors":"Brian Mcfee, Colin Raffel, Dawen Liang, P W Daniel, Matt Ellis, Eric Mcvicar, Oriol Battenberg, Nieto","year":"2015","venue":"Proc. Python in Science Conf","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Oquab et al., 2015] Maxime Oquab, Léon Bottou, Ivan Laptev, and Josef Sivic. Is object localization for free?- weakly-supervised learning with convolutional neural net- works. In Proc. Int. Computer Vision and Pattern Recog- nition, pages 685-694, 2015.","title":"Is object localization for free?weakly-supervised learning with convolutional neural networks","authors":"Maxime Oquab, Léon Bottou, Ivan Laptev, Josef Sivic","year":"2015","venue":"Proc. Int. Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Pachet, 2016] Franc ¸ois Pachet. A joyful ode to auto- matic orchestration. ACM Trans. Intell. Syst. Technol., 8(2):18:1-18:13, 2016.","title":"Franc ¸ois Pachet. A joyful ode to automatic orchestration","authors":"Pachet","year":"2016","venue":"ACM Trans. Intell. Syst. Technol","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Pati, 2018] Ashis Pati. Neural style transfer for musi- cal melodies, 2018. [Online] https://ashispati.github.io/ style-transfer/.","title":"Neural style transfer for musical melodies","authors":"Ashis Pati","year":"2018","venue":null,"doi":null,"url":"https://ashispati.github.io/style-transfer/","internal_url":null},{"position":28,"raw":"[Raffel and Ellis, 2016] Colin Raffel and Daniel P. W. El- lis. Optimizing DTW-based audio-to-MIDI alignment and matching. In Proc. IEEE Int. Conf. Acoustics, Speech and Signal Processing, pages 81-85, 2016.","title":"Optimizing DTW-based audio-to-MIDI alignment and matching","authors":"Colin Raffel, P W Daniel, Ellis","year":"2016","venue":"Proc. IEEE Int. Conf. Acoustics, Speech and Signal Processing","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Roberts et al., 2017] Adam Roberts, Jesse Engel, and Dou- glas Eck. Hierarchical variational autoencoders for music. In Proc. NIPS Workshop on Machine Learning for Cre- ativity and Design, 2017.","title":"Hierarchical variational autoencoders for music","authors":"Adam Roberts, Jesse Engel, Douglas Eck","year":"2017","venue":"Proc. NIPS Workshop on Machine Learning for Creativity and Design","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Ronneberger et al., 2015] Olaf Ronneberger, Philipp Fis- cher, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Proc. Int. Conf. Medical Image Computing and Computer-Assisted Inter- vention, pages 234-241. Springer, 2015.","title":"Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation","authors":"Ronneberger","year":"2015","venue":"Proc. Int. Conf. Medical Image Computing and Computer-Assisted Intervention","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}