{"type":"ScholarlyArticle","title":"Sound2Synth: Interpreting Sound via FM Synthesizer Parameters Estimation","authors":[{"name":"Zui Chen","slug":"zui-chen","given":"Zui","family":"Chen","affiliation":"Institute for Interdisciplinary Information Sciences , Tsinghua University"},{"name":"Yansen Jing","slug":"yansen-jing","given":"Yansen","family":"Jing","affiliation":"Institute for Interdisciplinary Information Sciences , Tsinghua University"},{"name":"Shengcheng Yuan","slug":"shengcheng-yuan","given":"Shengcheng","family":"Yuan","affiliation":"Beijing DeepMusic Technology Co. , Ltd"},{"name":"Yifei Xu","slug":"yifei-xu","given":"Yifei","family":"Xu","affiliation":"Beijing DeepMusic Technology Co. , Ltd"},{"name":"Jian Wu","slug":"jian-wu","given":"Jian","family":"Wu","affiliation":"Beijing DeepMusic Technology Co. , Ltd"},{"name":"Hang Zhao","slug":"hang-zhao","given":"Hang","family":"Zhao","affiliation":"Institute for Interdisciplinary Information Sciences , Tsinghua University"}],"abstract":"Synthesizer is a type of electronic musical instrument that is now widely used in modern music production and sound design. Each parameters configuration of a synthesizer produces a unique timbre and can be viewed as a unique instrument. The problem of estimating a set of parameters configuration that best restore a sound timbre is an important yet complicated problem, i.e.: the synthesizer parameters estimation problem. We proposed a multi-modal deep-learning-based pipeline Sound2Synth, together with a network structure Prime-Dilated Convolution (PDC) specially designed to solve this problem. Our method achieved not only SOTA but also the first real-world applicable results on Dexed synthesizer, a popular FM synthesizer.","doi":"10.24963/ijcai.2022/682","url":"https://ijcai.org/proceedings/2022/682","pdf":"https://ijcai.org/proceedings/2022/0682.pdf","markdown":"https://ijcai.org/proceedings/2022/682.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/682","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Research Papers","track":"AI and Arts","number":682,"pages":{"first":4921,"last":4928},"keywords":["Methods and resources: Machine learning, deep learning, neural models, reinforcement learning","Application domains: Music"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987162","length_sec":97,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985305","length_sec":691,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Barkan and Tsiris, 2018] Oren Barkan and David Tsiris. Deep synthesizer parameter estimation. CoRR, abs/1812.06349, 2018.","title":"Deep synthesizer parameter estimation","authors":"Oren Barkan, David Tsiris","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Burk, 2014] Phil Burk. jsyn. https://github.com/philburk/ jsyn, 2014. Accessed: 2022-06-18.","title":"Phil Burk. jsyn","authors":"Burk","year":"2014","venue":null,"doi":null,"url":"https://github.com/philburk/jsyn","internal_url":null},{"position":3,"raw":"[Engel et al., 2020] Jesse H. Engel, Lamtharn Hantrakul, Chenjie Gu, and Adam Roberts. DDSP: differentiable digital signal processing. In ICLR, volume abs/2001.04643, 2020.","title":"DDSP: differentiable digital signal processing","authors":"Jesse H Engel, Lamtharn Hantrakul, Chenjie Gu, Adam Roberts","year":"2020","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Esling et al., 2019] Philippe Esling, Naotake Masuda, Adrien Bardet, Romeo Despres, and Axel Chemla- Romeu-Santos. Universal audio synthesizer control with normalizing flows. CoRR, abs/1907.00971, 2019.","title":"Universal audio synthesizer control with normalizing flows","authors":"Philippe Esling, Naotake Masuda, Adrien Bardet, Romeo Despres, Axel Chemla-Romeu-Santos","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Gauthier, 2013] Pascal Gauthier. dexed. https://github.com/ asb2m10/dexed, 2013. Accessed: 2022-06-18.","title":null,"authors":"Pascal Gauthier","year":"2013","venue":null,"doi":null,"url":"https://github.com/asb2m10/dexed","internal_url":null},{"position":6,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR. IEEE, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR. IEEE","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9:1735-80, 12 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Horner et al., 1993] Andrew Horner, James Beauchamp, and Lippold Haken. Machine tongues xvi: Genetic algorithms and their application to fm matching synthesis. Computer Music Journal, 17(4):17-29, 1993.","title":"Machine tongues xvi: Genetic algorithms and their application to fm matching synthesis","authors":"Andrew Horner, James Beauchamp, Lippold Haken","year":"1993","venue":"Computer Music Journal","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Huang et al., 2016] Gao Huang, Zhuang Liu, and Kilian Q. Weinberger. Densely connected convolutional networks. CoRR, 2016.","title":"Densely connected convolutional networks","authors":"Gao Huang, Zhuang Liu, Kilian Q Weinberger","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Le Vaillant et al., 2021] Gwendal Le Vaillant, Thierry Du- toit, and Sébastien Dekeyser. Improving synthesizer pro- gramming from variational autoencoders latent space. In DAFx20in21, 2021.","title":"Improving synthesizer programming from variational autoencoders latent space","authors":"Le Gwendal, Thierry Vaillant, Sébastien Dutoit, Dekeyser","year":"2021","venue":"DAFx","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[LeCun et al., 1998] Yann LeCun, Léon Bottou, Yoshua Ben- gio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proc. IEEE, 86(11):2278-2324, 1998.","title":"Gradient-based learning applied to document recognition","authors":"Yann Lecun, Léon Bottou, Yoshua Bengio, Patrick Haffner","year":"1998","venue":"Proc. IEEE","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Loshchilov and Hutter, 2019] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.","title":"Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization","authors":"Hutter Loshchilov","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mitchell and Sullivan, 2005] Thomas Mitchell and Charlie Sullivan. Frequency modulation tone matching using a fuzzy clustering evolution strategy. In AES, 2005.","title":"Frequency modulation tone matching using a fuzzy clustering evolution strategy","authors":"Thomas Mitchell, Charlie Sullivan","year":"2005","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mitcheltree and Koike, 2021] Christopher Mitcheltree and Hideki Koike. Serumrnn: Step by step audio VST effect programming. CoRR, abs/2104.03876, 2021. [Records, 2014] Xfer Records. Serum: Advanced wavetable synthesizer -xfer records. https://xferrecords.com/products/ serum, 2014. Accessed: 2022-06-18.","title":"Serumrnn: Step by step audio VST effect programming","authors":"Koike Mitcheltree, Christopher Mitcheltree, Hideki Koike","year":"2014","venue":"Xfer Records. Serum: Advanced wavetable synthesizer -xfer records","doi":null,"url":"https://xferrecords.com/products/serum","internal_url":null},{"position":15,"raw":"[Roth and Yee-King, 2011] Martin Roth and Matthew Yee- King. A comparison of parametric optimization techniques for musical instrument tone matching. In AES, 2011.","title":"A comparison of parametric optimization techniques for musical instrument tone matching","authors":"Roth, Yee-King ;, Martin Roth, Matthew Yee-King","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Simonyan and Zisserman, 2015] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. In Yoshua Bengio and Yann LeCun, editors, ICLR, 2015.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Terasawa et al., 2005] Hiroko Terasawa, Malcolm Slaney, and Jonathan Berger. Perceptual distance in timbre space. In ICAD, 01 2005.","title":"Perceptual distance in timbre space","authors":"Hiroko Terasawa, Malcolm Slaney, Jonathan Berger","year":"2005","venue":"ICAD","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[u-he Software Synthesizers and Effects, 2011] u-he Soft- ware Synthesizers and Effects. Diva: The spirit of analogue -u-he. https://u-he.com/products/diva, 2011. Accessed: 2022-06-18.","title":"u-he Software Synthesizers and Effects. Diva: The spirit of analogue -u-he","authors":null,"year":"2011","venue":null,"doi":null,"url":"https://u-he.com/products/diva,2011","internal_url":null},{"position":19,"raw":"[Yee-King et al., 2018] Matthew John Yee-King, Leon Fed- den, and Mark d'Inverno. Automatic programming of VST sound synthesizers using deep networks and other tech- niques. IEEE Trans. Emerg. Top. Comput. Intell., 2(2):150- 159, 2018.","title":"Automatic programming of VST sound synthesizers using deep networks and other techniques","authors":"Matthew John, Yee-King, Leon Fedden, Mark D' Inverno","year":"2018","venue":"IEEE Trans. Emerg. Top. Comput. Intell","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}