{"type":"ScholarlyArticle","title":"NAS-FM: Neural Architecture Search for Tunable and Interpretable Sound Synthesis Based on Frequency Modulation","authors":[{"name":"Zhen Ye","slug":"zhen-ye","given":"Zhen","family":"Ye","affiliation":"Hong Kong Baptist University"},{"name":"Wei Xue","slug":"wei-xue","given":"Wei","family":"Xue","affiliation":"Hong Kong University of Science and Technology"},{"name":"Xu Tan","slug":"xu-tan","given":"Xu","family":"Tan","affiliation":"Microsoft Research Asia"},{"name":"Qifeng Liu","slug":"qifeng-liu","given":"Qifeng","family":"Liu","affiliation":"Hong Kong University of Science and Technology; Hong Kong Institute of Science & Innovation , Chinese Academy of Sciences"},{"name":"Yike Guo","slug":"yike-guo","given":"Yike","family":"Guo","affiliation":"Hong Kong University of Science and Technology"}],"abstract":"Developing digital sound synthesizers is crucial to the music industry as it provides a low-cost way to produce high-quality sounds with rich timbres. Existing traditional synthesizers often require substantial expertise to determine the overall framework of a synthesizer and the parameters of submodules. Since expert knowledge is hard to acquire,  it hinders the flexibility to quickly design and tune digital synthesizers for diverse sounds. In this paper, we propose ``NAS-FM'', which adopts neural architecture search (NAS) to build a differentiable frequency modulation (FM) synthesizer. Tunable synthesizers with interpretable controls can be developed automatically from sounds without any prior expert knowledge and manual operating costs. In detail, we train a supernet with a specifically designed search space, including predicting the envelopes of carriers and modulators with different frequency ratios. An evolutionary search algorithm with adaptive oscillator size is then developed to find the optimal relationship between oscillators and the frequency ratio of FM. Extensive experiments on recordings of different instrument sounds show that our algorithm can build a synthesizer fully automatically, achieving better results than handcrafted synthesizers. Audio samples are available at https://nas-fm.github.io/","doi":"10.24963/ijcai.2023/651","url":"https://ijcai.org/proceedings/2023/651","pdf":"https://ijcai.org/proceedings/2023/0651.pdf","markdown":"https://ijcai.org/proceedings/2023/651.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/651","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"AI and Arts","track":"AI and Arts","number":651,"pages":{"first":5869,"last":5877},"keywords":["Methods and resources: Machine learning, deep learning, neural models, reinforcement learning","Application domains: Music and sound"],"videos":[],"references":[{"position":1,"raw":"[Bai et al., 2018] Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01271, 2018.","title":"An empirical evaluation of generic convolutional and recurrent networks for sequence modeling","authors":"Bai","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bender et al., 2018] Gabriel Bender, Pieter-Jan Kinder- mans, Barret Zoph, Vijay Vasudevan, and Quoc Le. Un- derstanding and simplifying one-shot architecture search. In International conference on machine learning, pages 550-559. PMLR, 2018.","title":"Understanding and simplifying one-shot architecture search","authors":"Gabriel Bender, Pieter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, Quoc Le","year":"2018","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bristow-Johnson, 1996] Robert Bristow-Johnson. Wavetable synthesis 101, a fundamental perspective. In Audio engineering society convention 101. Audio Engineering Society, 1996.","title":"Wavetable synthesis 101, a fundamental perspective","authors":"Robert Bristow-Johnson","year":"1996","venue":"Audio engineering society convention 101","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Caspe et al., 2022] Franco Caspe, Andrew McPherson, and Mark Sandler. Ddx7: Differentiable fm synthesis of musi- cal instrument sounds. arXiv preprint arXiv:2208.06169, 2022.","title":"Ddx7: Differentiable fm synthesis of musical instrument sounds","authors":"Andrew Franco Caspe, Mark Mcpherson, Sandler","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2022] Zui Chen, Yansen Jing, Shengcheng Yuan, Yifei Xu, Jian Wu, and Hang Zhao. Sound2synth: Interpreting sound via fm synthesizer parameters estima- tion. arXiv preprint arXiv:2205.03043, 2022.","title":"Sound2synth: Interpreting sound via fm synthesizer parameters estimation","authors":"Zui Chen, Yansen Jing, Shengcheng Yuan, Yifei Xu, Jian Wu, Hang Zhao","year":"2022","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2022/682"},{"position":6,"raw":"[Chowning, 1973] John M Chowning. The synthesis of com- plex audio spectra by means of frequency modulation. Journal of the audio engineering society, 21(7):526-534, 1973.","title":"The synthesis of complex audio spectra by means of frequency modulation","authors":"M John, Chowning","year":"1973","venue":"Journal of the audio engineering society","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Engel et al., 2019] Jesse Engel, Kumar Krishna Agrawal, Shuo Chen, Ishaan Gulrajani, Chris Donahue, and Adam Roberts. Gansynth: Adversarial neural audio synthesis. arXiv preprint arXiv:1902.08710, 2019.","title":"Gansynth: Adversarial neural audio synthesis","authors":"Jesse Engel, Krishna Kumar, Shuo Agrawal, Ishaan Chen, Chris Gulrajani, Adam Donahue, Roberts","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Engel et al., 2020] Jesse Engel, Lamtharn Hantrakul, Chen- jie Gu, and Adam Roberts. Ddsp: Differentiable digital signal processing. arXiv preprint arXiv:2001.04643, 2020.","title":"Ddsp: Differentiable digital signal processing","authors":"Jesse Engel, Lamtharn Hantrakul, Chenjie Gu, Adam Roberts","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gauthier, 2013] Pascal Gauthier. dexed. https://github.com/ asb2m10/dexed, 2013. Accessed: 2023-06-07.","title":null,"authors":"Pascal Gauthier","year":"2013","venue":null,"doi":null,"url":"https://github.com/asb2m10/dexed","internal_url":null},{"position":10,"raw":"[Guo et al., 2020] Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, and Jian Sun. Single path one-shot neural architecture search with uni- form sampling. In European conference on computer vi- sion, pages 544-560. Springer, 2020.","title":"Single path one-shot neural architecture search with uniform sampling","authors":"Guo","year":"2020","venue":"European conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Horner et al., 1993] Andrew Horner, James Beauchamp, and Lippold Haken. Machine tongues xvi: Genetic al- gorithms and their application to fm matching synthesis. Computer Music Journal, 17(4):17-29, 1993.","title":"Machine tongues xvi: Genetic algorithms and their application to fm matching synthesis","authors":"Andrew Horner, James Beauchamp, Lippold Haken","year":"1993","venue":"Computer Music Journal","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Horner, 1996] Andrew Horner. Double-modulator fm matching of instrument tones. Computer Music Journal, 20(2):57-71, 1996.","title":"Double-modulator fm matching of instrument tones","authors":"Andrew Horner","year":"1996","venue":"Computer Music Journal","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Horner, 1998] Andrew Horner. Nested modulator and feed- back fm matching of instrument tones. IEEE Transactions on Speech and Audio Processing, 6(4):398-409, 1998.","title":"Nested modulator and feedback fm matching of instrument tones","authors":"Andrew Horner","year":"1998","venue":"IEEE Transactions on Speech and Audio Processing","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Huovilainen and Välimäki, 2005] Antti Huovilainen and Vesa Välimäki. New approaches to digital subtractive syn- thesis. In ICMC, 2005. [Jh, 1975] Holland Jh. Adaptation in natural and artificial systems. Ann Arbor, 1975.","title":"New approaches to digital subtractive synthesis","authors":"Välimäki Huovilainen, Antti Huovilainen, Vesa Välimäki","year":"1975","venue":"Holland Jh. Adaptation in natural and artificial systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Justice, 1979] James Justice. Analytic signal processing in music computation. IEEE Transactions on Acoustics, Speech, and Signal Processing, 27(6):670-684, 1979.","title":"Analytic signal processing in music computation","authors":"James Justice","year":"1979","venue":"IEEE Transactions on Acoustics, Speech, and Signal Processing","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kalchbrenner et al., 2018] Nal Kalchbrenner, Erich Elsen, Karen Simonyan, Seb Noury, Norman Casagrande, Ed- ward Lockhart, Florian Stimberg, Aaron Oord, Sander Dieleman, and Koray Kavukcuoglu. Efficient neural audio synthesis. In International Conference on Machine Learn- ing, pages 2410-2419. PMLR, 2018.","title":"Efficient neural audio synthesis","authors":"Nal Kalchbrenner, Erich Elsen, Karen Simonyan, Seb Noury, Norman Casagrande, Edward Lockhart, Florian Stimberg, Aaron Oord, Sander Dieleman, Koray Kavukcuoglu","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kilgour et al., 2018] Kevin Kilgour, Mauricio Zuluaga, Do- minik Roblek, and Matthew Sharifi. Fréchet audio dis- tance: A metric for evaluating music enhancement algo- rithms. arXiv preprint arXiv:1812.08466, 2018.","title":"Fréchet audio distance: A metric for evaluating music enhancement algorithms","authors":"Kevin Kilgour, Mauricio Zuluaga, Dominik Roblek, Matthew Sharifi","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kim et al., 2018] Jong Wook Kim, Justin Salamon, Peter Li, and Juan Pablo Bello. Crepe: A convolutional repre- sentation for pitch estimation. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 161-165. IEEE, 2018.","title":"Crepe: A convolutional representation for pitch estimation","authors":"Jong Wook, Kim, Justin Salamon, Peter Li, Juan Pablo Bello","year":"2018","venue":"IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Kong et al., 2020a] Jungil Kong, Jaehyeon Kim, and Jaeky- oung Bae. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in Neural Information Processing Systems, 33:17022-17033, 2020.","title":"Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis","authors":"Kong","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Kong et al., 2020b] Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versa- tile diffusion model for audio synthesis. arXiv preprint arXiv:2009.09761, 2020.","title":"Diffwave: A versatile diffusion model for audio synthesis","authors":"Kong","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Le Vaillant et al., 2021] Gwendal Le Vaillant, Thierry Du- toit, and Sébastien Dekeyser. Improving synthesizer pro- gramming from variational autoencoders latent space. In 2021 24th International Conference on Digital Audio Ef- fects (DAFx), pages 276-283. IEEE, 2021.","title":"Improving synthesizer programming from variational autoencoders latent space","authors":"Le Gwendal, Thierry Vaillant, Sébastien Dutoit, Dekeyser","year":"2021","venue":"24th International Conference on Digital Audio Effects (DAFx)","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Li et al., 2018] Bochen Li, Xinzhao Liu, Karthik Dinesh, Zhiyao Duan, and Gaurav Sharma. Creating a multitrack classical music performance dataset for multimodal mu- sic analysis: Challenges, insights, and applications. IEEE Transactions on Multimedia, 21(2):522-535, 2018.","title":"Creating a multitrack classical music performance dataset for multimodal music analysis: Challenges, insights, and applications","authors":"Li","year":"2018","venue":"IEEE Transactions on Multimedia","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Liang et al., 2021] Tingting Liang, Yongtao Wang, Zhi Tang, Guosheng Hu, and Haibin Ling. Opanas: One-shot path aggregation network architecture search for object de- tection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10195- 10203, 2021.","title":"Opanas: One-shot path aggregation network architecture search for object detection","authors":"Tingting Liang, Yongtao Wang, Zhi Tang, Guosheng Hu, Haibin Ling","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Liu et al., 2018] Hanxiao Liu, Karen Simonyan, and Yim- ing Yang. Darts: Differentiable architecture search. arXiv preprint arXiv:1806.09055, 2018.","title":"Darts: Differentiable architecture search","authors":"Liu","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Mehri et al., 2016] Soroush Mehri, Kundan Kumar, Ishaan Gulrajani, Rithesh Kumar, Shubham Jain, Jose Sotelo, Aaron Courville, and Yoshua Bengio. Samplernn: An unconditional end-to-end neural audio generation model. arXiv preprint arXiv:1612.07837, 2016.","title":"Samplernn: An unconditional end-to-end neural audio generation model","authors":"Soroush Mehri, Kundan Kumar, Ishaan Gulrajani, Rithesh Kumar, Shubham Jain, Jose Sotelo, Aaron Courville, Yoshua Bengio","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Mitchell and Sullivan, 2005] Thomas Mitchell and Charlie Sullivan. Frequency modulation tone matching using a fuzzy clustering evolution strategy. In Audio Engineer- ing Society Convention 118. Audio Engineering Society, 2005.","title":"Frequency modulation tone matching using a fuzzy clustering evolution strategy","authors":"Thomas Mitchell, Charlie Sullivan","year":"2005","venue":"Audio Engineering Society Convention 118","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Moore et al., 1997] Brian CJ Moore, Brian R Glasberg, and Thomas Baer. A model for the prediction of thresholds, loudness, and partial loudness. Journal of the Audio Engi- neering Society, 45(4):224-240, 1997.","title":"A model for the prediction of thresholds, loudness, and partial loudness","authors":"Brian Cj Moore, Brian R Glasberg, Thomas Baer","year":"1997","venue":"Journal of the Audio Engineering Society","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Oord et al., 2016] Aaron van den Oord, Sander Diele- man, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw au- dio. arXiv preprint arXiv:1609.03499, 2016.","title":"Wavenet: A generative model for raw audio","authors":"Oord","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Pham et al., 2018] Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. Efficient neural architecture search via parameters sharing. In International conference on machine learning, pages 4095-4104. PMLR, 2018.","title":"Efficient neural architecture search via parameters sharing","authors":"Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, Jeff Dean","year":"2018","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Prenger et al., 2019] Ryan Prenger, Rafael Valle, and Bryan Catanzaro. Waveglow: A flow-based generative network for speech synthesis. In ICASSP 2019-2019 IEEE Inter- national Conference on Acoustics, Speech and Signal Pro- cessing (ICASSP), pages 3617-3621. IEEE, 2019.","title":"Waveglow: A flow-based generative network for speech synthesis","authors":"Ryan Prenger, Rafael Valle, Bryan Catanzaro","year":"2019","venue":"ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Schottstaedt, 1977] Bill Schottstaedt. The simulation of nat- ural instrument tones using frequency modulation with a complex modulating wave. Computer Music Journal, pages 46-50, 1977.","title":"Bill Schottstaedt. The simulation of natural instrument tones using frequency modulation with a complex modulating wave","authors":"Schottstaedt","year":"1977","venue":"Computer Music Journal","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Serra and Smith, 1990] Xavier Serra and Julius Smith. Spectral modeling synthesis: A sound analysis/synthesis system based on a deterministic plus stochastic decompo- sition. Computer Music Journal, 14(4):12-24, 1990.","title":"Xavier Serra and Julius Smith. Spectral modeling synthesis: A sound analysis/synthesis system based on a deterministic plus stochastic decomposition","authors":"Smith Serra","year":"1990","venue":"Computer Music Journal","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Tang et al., 2020] Haotian Tang, Zhijian Liu, Shengyu Zhao, Yujun Lin, Ji Lin, Hanrui Wang, and Song Han. Searching efficient 3d architectures with sparse point- voxel convolution. In European conference on computer vision, pages 685-702. Springer, 2020.","title":"Searching efficient 3d architectures with sparse pointvoxel convolution","authors":"Tang","year":"2020","venue":"European conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Xu et al., 2021] Jin Xu, Xu Tan, Renqian Luo, Kaitao Song, Jian Li, Tao Qin, and Tie-Yan Liu. Nas-bert: task-agnostic and adaptive-size bert compression with neural architec- ture search. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 1933-1943, 2021.","title":"Nas-bert: task-agnostic and adaptive-size bert compression with neural architecture search","authors":"Jin Xu, Xu Tan, Renqian Luo, Kaitao Song, Jian Li, Tao Qin, Tie-Yan Liu","year":"2021","venue":"Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Xu et al., 2022] Jin Xu, Xu Tan, Kaitao Song, Renqian Luo, Yichong Leng, Tao Qin, Tie-Yan Liu, and Jian Li. An- alyzing and mitigating interference in neural architecture search. In International Conference on Machine Learn- ing, pages 24646-24662. PMLR, 2022.","title":"Analyzing and mitigating interference in neural architecture search","authors":"Jin Xu, Xu Tan, Kaitao Song, Renqian Luo, Yichong Leng, Tao Qin, Tie-Yan Liu, Jian Li","year":"2022","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Yee-King et al., 2018] Matthew John Yee-King, Leon Fed- den, and Mark d'Inverno. Automatic programming of vst sound synthesizers using deep networks and other tech- niques. IEEE Transactions on Emerging Topics in Com- putational Intelligence, 2(2):150-159, 2018.","title":"Automatic programming of vst sound synthesizers using deep networks and other techniques","authors":"Matthew John, Yee-King, Leon Fedden, Mark D' Inverno","year":"2018","venue":"IEEE Transactions on Emerging Topics in Computational Intelligence","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}