{"type":"ScholarlyArticle","title":"Seq-U-Net: A One-Dimensional Causal U-Net for Efficient Sequence Modelling","authors":[{"name":"Daniel Stoller","slug":"daniel-stoller","given":"Daniel","family":"Stoller","affiliation":"Queen Mary University of London; Statistics and Machine Learning ,"},{"name":"Mi Tian","slug":"mi-tian","given":"Mi","family":"Tian","affiliation":"Spotify"},{"name":"Sebastian Ewert","slug":"sebastian-ewert","given":"Sebastian","family":"Ewert","affiliation":"Spotify"},{"name":"Simon Dixon","slug":"simon-dixon","given":"Simon","family":"Dixon","affiliation":"Queen Mary University of London"}],"abstract":"Convolutional neural networks (CNNs) with dilated filters such as the Wavenet or the Temporal Convolutional Network (TCN) have shown good results in a variety of sequence modelling tasks. While their receptive field grows exponentially with the number of layers, computing the convolutions over very long sequences of features in each layer is time and memory-intensive, and prohibits the use of longer receptive fields in practice. To increase efficiency, we make use of the \"slow feature\" hypothesis stating that many features of interest are slowly varying over time. For this, we use a U-Net architecture that computes features at multiple time-scales and adapt it to our auto-regressive scenario by making convolutions causal. We apply our model (\"Seq-U-Net\") to a variety of tasks including language and audio generation. In comparison to TCN and Wavenet, our network consistently saves memory and computation time, with speed-ups for training and inference of over 4x in the audio generation experiment in particular, while achieving a comparable performance on real-world tasks.","doi":"10.24963/ijcai.2020/400","url":"https://ijcai.org/proceedings/2020/400","pdf":"https://ijcai.org/proceedings/2020/0400.pdf","markdown":"https://ijcai.org/proceedings/2020/400.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/400","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":400,"pages":{"first":2893,"last":2900},"keywords":["Machine Learning: Deep Learning: Convolutional networks","Machine Learning: Deep Learning: Sequence Modeling"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25522/file_25522.m3u8","length_sec":250,"provider":"hls"}],"references":[{"position":1,"raw":"[Allan and Williams, 2005] Moray Allan and Christopher Williams. Harmonising Chorales by Probabilistic Infer- ence. In L. K. Saul, Y. Weiss, and L. Bottou, editors, Advances in Neural Information Processing Systems 17, pages 25-32. MIT Press, 2005.","title":"Moray Allan and Christopher Williams. Harmonising Chorales by Probabilistic Inference","authors":"Williams Allan","year":"2005","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bai et al., 2018] Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. Convolutional Sequence Modeling Revisited. In International Conference on Learning Representations (ICLR) Workshop Track, 2018.","title":"Convolutional Sequence Modeling Revisited","authors":"J Zico Shaojie Bai, Vladlen Kolter, Koltun","year":"2018","venue":"International Conference on Learning Representations (ICLR) Workshop Track","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bengio et al., 1994] Y. Bengio, P. Simard, and P. Frasconi. Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5(2):157-166, 1994.","title":"Learning long-term dependencies with gradient descent is difficult","authors":"Y Bengio, P Simard, P Frasconi","year":"1994","venue":"IEEE Transactions on Neural Networks","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Boulanger-Lewandowski et al., 2012] Nicolas Boulanger- Lewandowski, Yoshua Bengio, and Pascal Vincent. Modeling temporal dependencies in high-dimensional sequences: Application to polyphonic music generation and transcription. Proc. of the International Conference on Machine Learning (ICML), pages 1159-1166, 2012.","title":"Modeling temporal dependencies in high-dimensional sequences: Application to polyphonic music generation and transcription","authors":"Boulanger-Lewandowski","year":"2012","venue":"Proc. of the International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Campos et al., 2017] Victor Campos, Brendan Jou, Xavier Giro-i-Nieto, Jordi Torres, and Shih-Fu Chang. Skip RNN: Learning to skip state updates in recurrent neural net- works. CoRR, abs/1708.06834, 2017.","title":"Skip RNN: Learning to skip state updates in recurrent neural networks","authors":"Victor Campos, Brendan Jou, Xavier Giro-I-Nieto, Jordi Torres, Shih-Fu Chang","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chang et al., 2017] Shiyu Chang, Yang Zhang, Wei Han, Mo Yu, Xiaoxiao Guo, Wei Tan, Xiaodong Cui, Michael Witbrock, Mark A Hasegawa-Johnson, and Thomas S Huang. Dilated Recurrent Neural Networks. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vish- wanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 77-87.","title":"Dilated Recurrent Neural Networks","authors":"Shiyu Chang, Yang Zhang, Wei Han, Mo Yu, Xiaoxiao Guo, Wei Tan, Xiaodong Cui, Michael Witbrock, Mark A Hasegawa-Johnson, Thomas S Huang","year":null,"venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"Curran Associates, Inc., 2017.","title":null,"authors":"Curran Associates, Inc","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Child et al., 2019] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating Long Sequences with Sparse Transformers. CoRR, abs/1904.10509, 2019.","title":"Generating Long Sequences with Sparse Transformers","authors":"Child","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Chung et al., 2016] Junyoung Chung, Sungjin Ahn, and Yoshua Bengio. Hierarchical Multiscale Recurrent Neu- ral Networks. CoRR, abs/1609.01704, 2016.","title":"Hierarchical Multiscale Recurrent Neural Networks","authors":"Chung","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Dieleman et al., 2018] Sander Dieleman, Aäron van den Oord, and Karen Simonyan. The challenge of realistic music generation: Modelling raw audio at scale. In Ad- vances in Neural Information Processing Systems 31: An- nual Conference on Neural Information Processing Sys- tems (NeurIPS), pages 8000-8010, 2018.","title":"The challenge of realistic music generation: Modelling raw audio at scale","authors":"Aäron Sander Dieleman, Karen Van Den Oord, Simonyan","year":"2018","venue":"Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems (NeurIPS)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[El Hihi and Bengio, 1995] Salah El Hihi and Yoshua Ben- gio. Hierarchical Recurrent Neural Networks for Long- term Dependencies. In Proceedings of the 8th Interna- tional Conference on Neural Information Processing Sys- tems, NIPS'95, pages 493-499, Denver, Colorado, 1995. MIT Press.","title":"Salah El Hihi and Yoshua Bengio. Hierarchical Recurrent Neural Networks for Longterm Dependencies","authors":"El Hihi, Bengio","year":"1995","venue":"Proceedings of the 8th International Conference on Neural Information Processing Systems, NIPS'95","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Graves et al., 2013] Alan Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. Speech recognition with deep recur- rent neural networks. In Proc. of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6645-6649, 2013.","title":"Speech recognition with deep recurrent neural networks","authors":"Alan Graves, Abdel-Rahman Mohamed, Geoffrey Hinton","year":"2013","venue":"Proc. of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[He et al., 2015] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CoRR, abs/1512.03385, 2015.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Huszár, 2015] Ferenc Huszár. How (not) to Train your Gen- erative Model: Scheduled Sampling, Likelihood, Adver- sary? CoRR, abs/1511.05101, 2015.","title":"Ferenc Huszár. How (not) to Train your Generative Model: Scheduled Sampling, Likelihood, Adversary? CoRR","authors":"Huszár","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Jin et al., 2018] Zeyu Jin, Adam Finkelstein, Gautham J. Mysore, and Jingwan Lu. FFTNet: A Real-Time Speaker- Dependent Neural Vocoder. In Proc. of the IEEE Inter- national Conference on Acoustics, Speech and Signal Pro- cessing (ICASSP), 2018.","title":"FFTNet: A Real-Time Speaker-Dependent Neural Vocoder","authors":"Jin","year":"2018","venue":"Proc. of the IEEE International Conference on Acoustics, Speech and Signal Processing","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kalchbrenner et al., 2016] Nal Kalchbrenner, Lasse Espe- holt, Karen Simonyan, Aaron van den Oord, Alex Graves, and Koray Kavukcuoglu. Neural Machine Translation in Linear Time. CoRR, abs/1610.10099, 2016.","title":"Neural Machine Translation in Linear Time","authors":"Nal Kalchbrenner, Lasse Espeholt, Karen Simonyan, Aaron Van Den Oord, Alex Graves, Koray Kavukcuoglu","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kalchbrenner et al., 2018] Nal Kalchbrenner, Erich Elsen, Karen Simonyan, Seb Noury, Norman Casagrande, Ed- ward Lockhart, Florian Stimberg, Aaron van den Oord, Sander Dieleman, and Koray Kavukcuoglu. Efficient Neu- ral Audio Synthesis. In Jennifer Dy and Andreas Krause, editors, Proc. of the International Conference on Machine Learning (ICML), volume 80, pages 2410-2419, Stock- holmsmässan, Stockholm Sweden, 2018. PMLR.","title":"Efficient Neural Audio Synthesis","authors":"Nal Kalchbrenner, Erich Elsen, Karen Simonyan, Seb Noury, Norman Casagrande, Edward Lockhart, Florian Stimberg, Aaron Van Den, Sander Oord, Koray Dieleman, Kavukcuoglu","year":"2018","venue":"Proc. of the International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Koutník et al., 2014] Jan Koutník, Klaus Greff, Faustino Gomez, and Jürgen Schmidhuber. A Clockwork RNN. CoRR, abs/1402.3511, February 2014.","title":"A Clockwork RNN","authors":"Koutník","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Marcus et al., 1993] Mitchell P. Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. Building a Large Annotated Corpus of English: The Penn Treebank. Comput. Linguist., 19(2):313-330, June 1993.","title":"Building a Large Annotated Corpus of English: The Penn Treebank","authors":"Mitchell P Marcus, Mary, Ann Marcinkiewicz, Beatrice Santorini","year":"1993","venue":"Comput. Linguist","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Mehri et al., 2016] Soroush Mehri, Kundan Kumar, Ishaan Gulrajani, Rithesh Kumar, Shubham Jain, Jose Sotelo, Aaron Courville, and Yoshua Bengio. SampleRNN: An Unconditional End-to-End Neural Audio Generation Model. CoRR, abs/1612.07837, 2016.","title":"SampleRNN: An Unconditional End-to-End Neural Audio Generation Model","authors":"Soroush Mehri, Kundan Kumar, Ishaan Gulrajani, Rithesh Kumar, Shubham Jain, Jose Sotelo, Aaron Courville, Yoshua Bengio","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Neil et al., 2016] Daniel Neil, Michael Pfeiffer, and Shih- Chii Liu. Phased LSTM: Accelerating Recurrent Net- work Training for Long or Event-based Sequences. CoRR, abs/1610.09513, 2016.","title":"Phased LSTM: Accelerating Recurrent Network Training for Long or Event-based Sequences","authors":"Daniel Neil, Michael Pfeiffer, Shih-Chii Liu","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Ronneberger et al., 2015] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In Proc. of the International Conference on Medical Image Computing and Computer- Assisted Intervention, pages 234-241. Springer, 2015.","title":"U-net: Convolutional networks for biomedical image segmentation","authors":"O Ronneberger, P Fischer, T Brox","year":"2015","venue":"Proc. of the International Conference on Medical Image Computing and Computer-Assisted Intervention","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Shelhamer et al., 2016] Evan Shelhamer, Kate Rakelly, Judy Hoffman, and Trevor Darrell. Clockwork Con- vnets for Video Semantic Segmentation. CoRR, abs/1608.03609, 2016.","title":"Clockwork Convnets for Video Semantic Segmentation","authors":"Evan Shelhamer, Kate Rakelly, Judy Hoffman, Trevor Darrell","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Stoller et al., 2018] Daniel Stoller, Sebastian Ewert, and Si- mon Dixon. Wave-U-Net: A Multi-Scale Neural Network for End-to-End Source Separation. In Proc. of the Interna- tional Society for Music Information Retrieval Conference (ISMIR), volume 19, pages 334-340, 2018.","title":"Wave-U-Net: A Multi-Scale Neural Network for End-to-End Source Separation","authors":"Daniel Stoller, Sebastian Ewert, Simon Dixon","year":"2018","venue":"Proc. of the International Society for Music Information Retrieval Conference (ISMIR)","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Trinh et al., 2018] Trieu H. Trinh, Andrew M. Dai, Minh- Thang Luong, and Quoc V. Le. Learning Longer-term Dependencies in RNNs with Auxiliary Losses. CoRR, abs/1803.00144, 2018.","title":"Learning Longer-term Dependencies in RNNs with Auxiliary Losses","authors":"H Trieu, Andrew M Trinh, Minh-Thang Dai, Luong, V Quoc, Le","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[van den Oord et al., 2016] Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Ko- ray Kavukcuoglu. WaveNet: A Generative Model for Raw Audio. arXiv:1609.03499, 2016.","title":"WaveNet: A Generative Model for Raw Audio","authors":"Oord Van Den","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[van den Oord et al., 2017] Aaron van den Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Ko- ray Kavukcuoglu, George van den Driessche, Edward Lockhart, Luis C. Cobo, Florian Stimberg, Norman Casagrande, Dominik Grewe, Seb Noury, Sander Diele- man, Erich Elsen, Nal Kalchbrenner, Heiga Zen, Alex Graves, Helen King, Tom Walters, Dan Belov, and Demis Hassabis. Parallel WaveNet: Fast High-Fidelity Speech Synthesis. CoRR, abs/1711.10433, 2017.","title":"Parallel WaveNet: Fast High-Fidelity Speech Synthesis","authors":"Oord Van Den","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Vasquez and Lewis, 2019] Sean Vasquez and Mike Lewis. MelNet: A Generative Model for Audio in the Frequency Domain. CoRR, abs/1906.01083, 2019.","title":"MelNet: A Generative Model for Audio in the Frequency Domain","authors":"Sean Vasquez, Mike Lewis","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. CoRR, abs/1706.03762, 2017.","title":"Attention Is All You Need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, Illia Polosukhin","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wiskott and Sejnowski, 2002] Laurenz Wiskott and Ter- rence J. Sejnowski. Slow Feature Analysis: Unsupervised Learning of Invariances. Neural Computation, 14(4):715- 770, 2002.","title":"Slow Feature Analysis: Unsupervised Learning of Invariances","authors":"Laurenz Wiskott, Terrence J Sejnowski","year":"2002","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Yu et al., 2017] Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient. In AAAI Conference on Ar- tificial Intelligence, pages 2852-2858, 2017.","title":"SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient","authors":"Lantao Yu, Weinan Zhang, Jun Wang, Yong Yu","year":"2017","venue":"AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}