{"type":"ScholarlyArticle","title":"Dilated Convolution with Dilated GRU for Music Source Separation","authors":[{"name":"Jen-Yu Liu","slug":"jen-yu-liu","given":"Jen-Yu","family":"Liu","affiliation":"Research Center for IT Innovation , Academia Sinica"},{"name":"Yi-Hsuan Yang","slug":"yi-hsuan-yang","given":"Yi-Hsuan","family":"Yang","affiliation":"Research Center for IT Innovation , Academia Sinica"}],"abstract":"Stacked dilated convolutions used in Wavenet have been shown effective for generating high-quality audios. By replacing pooling/striding with dilation in convolution layers, they can preserve high-resolution information and still reach distant locations. Producing high-resolution predictions is also crucial in music source separation, whose goal is to separate different sound sources while maintain the quality of the separated sounds. Therefore, in this paper, we use stacked dilated convolutions as the backbone for music source separation. Although stacked dilated convolutions can reach wider context than standard convolutions do, their effective receptive fields are still fixed and might not be wide enough for complex music audio signals. To reach even further information at remote locations, we propose to combine a dilated convolution with a modified GRU called Dilated GRU to form a block. A Dilated GRU receives information from k-step before instead of the previous step for a fixed k. This modification allows a GRU unit to reach a location with fewer recurrent steps and run faster because it can execute in parallel partially. We show that the proposed model with a stack of such blocks performs equally well or better than the state-of-the-art for separating both vocals and accompaniment.","doi":"10.24963/ijcai.2019/655","url":"https://ijcai.org/proceedings/2019/655","pdf":"https://ijcai.org/proceedings/2019/0655.pdf","markdown":"https://ijcai.org/proceedings/2019/655.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/655","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Multidisciplinary Topics and Applications","track":"Main track","number":655,"pages":{"first":4718,"last":4724},"keywords":["Multidisciplinary Topics and Applications: Art and Music","Machine Learning Applications: Applications of Supervised Learning","Machine Learning: Deep Learning"],"videos":[],"references":[{"position":1,"raw":"[Chang et al., 2017] Shiyu Chang, Yang Zhang, Wei Han, Mo Yu, Xiaoxiao Guo, Wei Tan, Xiaodong Cui, Michael Witbrock, Mark Hasegawa-Johnson, and Thomas S. Huang. Dilated recurrent neural networks. In Proc. Ad- vances in Neural Information Processing Systems, 2017.","title":"Dilated recurrent neural networks","authors":"Shiyu Chang, Yang Zhang, Wei Han, Mo Yu, Xiaoxiao Guo, Wei Tan, Xiaodong Cui, Michael Witbrock, Mark Hasegawa-Johnson, Thomas S Huang","year":"2017","venue":"Proc. Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cho et al., 2014] Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. On the proper- ties of neural machine translation: Encoder-decoder ap- proaches. In Proc. Workshop on Syntax, Semantics and Structure in Statistical Translation, 2014.","title":"On the properties of neural machine translation: Encoder-decoder approaches","authors":"Cho","year":"2014","venue":"Proc. Workshop on Syntax, Semantics and Structure in Statistical Translation","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chung et al., 2014] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evalua- tion of gated recurrent neural networks on sequence mod- eling. Proc. NIPS Workshop on Deep Learning, 2014.","title":"Empirical evalua-tion of gated recurrent neural networks on sequence modeling","authors":"Chung","year":"2014","venue":"Proc. NIPS Workshop on Deep Learning","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Howard et al., 2017] Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mo- bileNets: Efficient convolutional neural networks for mo- bile vision applications. arXiv preprint, apr 2017.","title":"Mo-bileNets: Efficient convolutional neural networks for mobile vision applications","authors":"Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, Hartwig Adam","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Jansson et al., 2017] Andreas Jansson, Eric Humphrey, Nicola Montecchio, Rachel Bittner, Aparna Kumar, and Tillman Weyde. Singing voice separation with deep U-net convolutional networks. In Proc. Int. Society for Music Information Retrieval Conf., 2017.","title":"Singing voice separation with deep U-net convolutional networks","authors":"Andreas Jansson, Eric Humphrey, Nicola Montecchio, Rachel Bittner, Aparna Kumar, Tillman Weyde","year":"2017","venue":"Proc. Int. Society for Music Information Retrieval Conf","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Kalchbrenner et al., 2017] Nal Kalchbrenner, Aäron van den Oord, Karen Simonyan, Ivo Danihelka, Oriol Vinyals, Alex Graves, and Koray Kavukcuoglu. Video pixel networks. In Proc. Int. Conf. Machine Learning, pages 1771-1779, 2017.","title":"Video pixel networks","authors":"Nal Kalchbrenner, Aäron Van Den Oord, Karen Simonyan, Ivo Danihelka, Oriol Vinyals, Alex Graves, Koray Kavukcuoglu","year":"2017","venue":"Proc. Int. Conf. Machine Learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Kingma and Ba, 2015] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. 2015.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Liu and Yang, 2018] Jen-Yu Liu and Yi-Hsuan Yang. De- noising auto-encoder with recurrent skip connections and residual regression for music source separation. In Proc. IEEE Int. Conf. Machine Learning and Applications, pages 773-778, 2018.","title":"Denoising auto-encoder with recurrent skip connections and residual regression for music source separation","authors":"Jen-Yu Liu, Yi-Hsuan Yang","year":"2018","venue":"Proc. IEEE Int. Conf. Machine Learning and Applications","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Liutkus et al., 2017] Antoine Liutkus, Fabian Robert Stöter, Zafar Rafii, Daichi Kitamura, Bertrand Rivet, Nobutaka Ito, Nobutaka Ono, and Julie Fontecave. The 2016 signal separation evaluation campaign. In Proc. LVA/ICA, pages 323-332. 2017.","title":"The 2016 signal separation evaluation campaign","authors":"Antoine Liutkus, Fabian Robert Stöter, Zafar Rafii, Daichi Kitamura, Bertrand Rivet, Nobutaka Ito, Nobutaka Ono, Julie Fontecave","year":"2017","venue":"Proc. LVA/ICA","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Mimilakis et al., 2018] Stylianos Ioannis Mimilakis, Kon- stantinos Drossos, Joao F. Santos, Gerald Schuller, Tuo- mas Virtanen, and Yoshua Bengio. Monaural singing voice separation with skip-filtering connections and recurrent in- ference of time-frequency mask. In IEEE Int. Conf. Acous- tics, Speech and Signal Processing, pages 721-725, 2018.","title":"Monaural singing voice separation with skip-filtering connections and recurrent inference of time-frequency mask","authors":"Stylianos Ioannis Mimilakis, Konstantinos Drossos, Joao F Santos, Gerald Schuller, Tuomas Virtanen, Yoshua Bengio","year":"2018","venue":"IEEE Int. Conf. Acoustics, Speech and Signal Processing","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Nugraha et al., 2016] Aditya Arie Nugraha, Antoine Li- utkus, and Emmanuel Vincent. Multichannel music sep- aration with deep neural networks. In Proc. European Sig- nal Processing Conf., pages 1748-1752, 2016.","title":"Multichannel music separation with deep neural networks","authors":"Arie Aditya, Antoine Nugraha, Emmanuel Liutkus, Vincent","year":"2016","venue":"Proc. European Signal Processing Conf","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Paulus and Virtanen, 2005] Jouni Paulus and Tuomas Virta- nen. Drum transcription with non-negative spectrogram factorisation. In Proc. European Signal Processing Conf., pages 1-4, 2005.","title":"Drum transcription with non-negative spectrogram factorisation","authors":"Virtanen Paulus, Jouni Paulus, Tuomas Virtanen","year":"2005","venue":"Proc. European Signal Processing Conf","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Rafii et al., 2018] Zafar Rafii, Antoine Liutkus, Fabian- Robert Stoter, Stylianos Ioannis Mimilakis, Derry FitzGer- ald, and Bryan Pardo. An overview of lead and accom- paniment separation in music. IEEE/ACM Trans. Audio, Speech and Lang. Proc., 26(8):1307-1335, August 2018.","title":"An overview of lead and accompaniment separation in music","authors":"Zafar Rafii, Antoine Liutkus, Fabian-Robert Stoter, Stylianos Ioannis Mimilakis, Derry Fitzgerald, Bryan Pardo","year":"2018","venue":"IEEE/ACM Trans. Audio, Speech and Lang. Proc","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Roma et al., 2018] Gerard Roma, Owen Green, and Pierre Alexandre Tremblay. Improving single-network single-channel separation of musical audio with con- volutional layers. In Proc. LVA/ICA, pages 306-315, 2018.","title":"Improving single-network single-channel separation of musical audio with convolutional layers","authors":"Roma","year":"2018","venue":"Proc. LVA/ICA","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Salimans and Kingma, 2016] Tim Salimans and Diederik P. Kingma. Weight normalization: A simple reparameter- ization to accelerate training of deep neural networks. Proc. Advances in Neural Information Processing Sys- tems, pages 901-909, 2016.","title":"Weight normalization: A simple reparameterization to accelerate training of deep neural networks","authors":"Kingma ; Salimans, Tim Salimans, Diederik P Kingma","year":"2016","venue":"Proc. Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Sercu and Goel, 2016] Tom Sercu and Vaibhava Goel. Dense prediction on sequences with time-dilated convo- lutions for speech recognition. arXiv preprint, nov 2016.","title":"Dense prediction on sequences with time-dilated convolutions for speech recognition","authors":"Goel Sercu, Tom Sercu, Vaibhava Goel","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Stoller et al., 2018] Daniel Stoller, Sebastian Ewert, and Si- mon Dixon. Wave-U-Net: A multi-scale neural network for end-to-end audio source separation. In Proc. Int. Soci- ety for Music Information Retrieval Conf., pages 334-340, 2018.","title":"Wave-U-Net: A multi-scale neural network for end-to-end audio source separation","authors":"Daniel Stoller, Sebastian Ewert, Simon Dixon","year":"2018","venue":"Proc. Int. Society for Music Information Retrieval Conf","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Stöter et al., 2018] Fabian-Robert Stöter, Antoine Liutkus, and Nobutaka Ito. The 2018 signal separation evaluation campaign. In Proc. LVA/ICA, pages 293-305, 2018.","title":"The 2018 signal separation evaluation campaign","authors":"Fabian-Robert Stöter, Antoine Liutkus, Nobutaka Ito","year":"2018","venue":"Proc. LVA/ICA","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Takahashi and Mitsufuji, 2017] Naoya Takahashi and Yuki Mitsufuji. Multi-scale multi-band densenets for audio source separation. In Proc. IEEE Workshop on Applica- tions of Signal Processing to Audio and Acoustics, pages 21-25, 2017.","title":"Multi-scale multi-band densenets for audio source separation","authors":"Naoya Takahashi, Yuki Mitsufuji","year":"2017","venue":"Proc. IEEE Workshop on Applications of Signal Processing to Audio and Acoustics","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Takahashi et al., 2018] Naoya Takahashi, Nabarun Goswami, and Yuki Mitsufuji. MMDenseLSTM: An efficient combination of convolutional and recurrent neural networks for audio source separation. Proc. International Workshop on Acoustic Signal Enhancement (IWAENC), pages 106-110, 2018.","title":"MMDenseLSTM: An efficient combination of convolutional and recurrent neural networks for audio source separation","authors":"Naoya Takahashi, Nabarun Goswami, Yuki Mitsufuji","year":"2018","venue":"Proc. International Workshop on Acoustic Signal Enhancement (IWAENC)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Uhlich et al., 2017] Stefan Uhlich, Marcello Porcu, Franck Giron, Michael Enenkl, Thomas Kemp, Naoya Takahashi, and Yuki Mitsufuji. Improving music source separation based on deep neural networks through data augmentation and network blending. In Proc. IEEE Int. Conf. Acoustics, Speech and Signal Processing, pages 261-265, 2017.","title":"Improving music source separation based on deep neural networks through data augmentation and network blending","authors":"Stefan Uhlich, Marcello Porcu, Franck Giron, Michael Enenkl, Thomas Kemp, Naoya Takahashi, Yuki Mitsufuji","year":"2017","venue":"Proc. IEEE Int. Conf. Acoustics, Speech and Signal Processing","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[van den Oord et al., 2016] Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Ko- ray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint, sep 2016.","title":"Wavenet: A generative model for raw audio","authors":"Oord Van Den","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Vande Veire and De Bie, 2018] Len Vande Veire and Tijl De Bie. From raw audio to a seamless mix: creating an automated DJ system for drum and bass. EURASIP J. Au- dio, Speech, and Music Processing, 2018(1):13, 2018.","title":"From raw audio to a seamless mix: creating an automated DJ system for drum and bass","authors":"Vande Veire, De Bie, ; ] Len, Vande Veire, Tijl De, Bie","year":"2018","venue":"EURASIP J. Audio, Speech, and Music Processing","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Vezhnevets et al., 2017] Alexander Sasha Vezhnevets, Si- mon Osindero, Tom Schaul, Nicolas Heess, Max Jader- berg, David Silver, and Koray Kavukcuoglu. FeUdal net- works for hierarchical reinforcement learning. In Proc. Int. Conf. Machine Learning, pages 3540-3549, 2017.","title":"FeUdal networks for hierarchical reinforcement learning","authors":"Sasha Alexander, Simon Vezhnevets, Tom Osindero, Nicolas Schaul, Max Heess, David Jaderberg, Koray Silver, Kavukcuoglu","year":"2017","venue":"Proc. Int. Conf. Machine Learning","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yu and Koltun, 2016] Fisher Yu and Vladlen Koltun. Multi- scale context aggregation by dilated convolutions. Proc. International Conference on Learning Representations (ICLR), 2016.","title":"Multiscale context aggregation by dilated convolutions","authors":"Koltun ; Fisher Yu, Vladlen Yu, Koltun","year":"2016","venue":"Proc. International Conference on Learning Representations (ICLR)","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhang et al., 2018] Xiangyu Zhang, Xinyu Zhou, Mengx- iao Lin, and Jian Sun. ShuffleNet: An extremely effi- cient convolutional neural network for mobile devices. In Proc. IEEE Conf. Computer Vision and Pattern Recogni- tion, 2018.","title":"ShuffleNet: An extremely efficient convolutional neural network for mobile devices","authors":"Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, Jian Sun","year":"2018","venue":"Proc. IEEE Conf. Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}