{"type":"ScholarlyArticle","title":"FlowSynth: Simplifying Complex Audio Generation Through Explorable Latent Spaces with Normalizing Flows","authors":[{"name":"Philippe Esling","slug":"philippe-esling","given":"Philippe","family":"Esling","affiliation":"IRCAM -CNRS UMR 9912 , Sorbonne Université -1 , place Igor Stravinsky 75004 Paris"},{"name":"Naotake Masuda","slug":"naotake-masuda","given":"Naotake","family":"Masuda","affiliation":"University of Tokyo , Tokyo , Japan"},{"name":"Axel Chemla--Romeu-Santos","slug":"axel-chemla-romeu-santos","given":"Axel","family":"Chemla--Romeu-Santos","affiliation":"IRCAM -CNRS UMR 9912 , Sorbonne Université -1 , place Igor Stravinsky 75004 Paris"}],"abstract":"Audio synthesizers are pervasive in modern music production. These highly complex audio generation functions provide a unique diversity through their large sets of parameters. However, this feature also can make them extremely hard and obfuscated to use, especially for non-expert users with no formal knowledge on signal processing.\r\nWe recently introduced a novel formalization of the problem of synthesizer control as learning an invertible mapping between an audio latent space, extracted from the audio signal, and a target parameter latent space, extracted from the synthesizer's presets, using normalizing flows. In addition to model a continuous representation allowing to ease the intuitive exploration of the synthesizer, it also provides a ground-breaking method for audio-based parameter inference, vocal control and macro-control learning.\r\nHere, we discuss the details of integrating these high-level features to develop new interaction schemes between a human user and the generating device: parameters inference from audio, high-level preset visualization and interpolation, that can be used both in off-time and real-time situations. Moreover, we also leverage LeapMotion devices to allow the control of hundreds of parameters simply by moving one hand across space to explore the low-dimensional latent space, allowing to both empower and facilitate the user's interaction with the synthesizer.","doi":"10.24963/ijcai.2020/767","url":"https://ijcai.org/proceedings/2020/767","pdf":"https://ijcai.org/proceedings/2020/0767.pdf","markdown":"https://ijcai.org/proceedings/2020/767.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/767","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Demos","track":"Demos","number":767,"pages":{"first":5273,"last":5275},"keywords":["Knowledge Representation and Reasoning: general","Machine Learning: general","Human-Computer Interactive Systems: general"],"videos":[],"references":[{"position":1,"raw":"[Arfib et al., 2002] Daniel Arfib, Jean-Michel Couturier, Loic Kessous, and Vincent Verfaille. Strategies of map- ping between gesture data and synthesis model parameters using perceptual spaces. Organised Sound, 7(2):127-144, 2002.","title":"Strategies of mapping between gesture data and synthesis model parameters using perceptual spaces","authors":"Daniel Arfib, Jean-Michel Couturier, Loic Kessous, Vincent Verfaille","year":"2002","venue":"Organised Sound","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Esling et al., 2020] Philippe Esling, Naotake Masuda, Adrien Bardet, Romeo Despres, and Axel Chemla- Romeu-Santos. Flow synthesizer: Universal audio synthesizer control with normalizing flows. Applied Sciences, 10(1):302, 2020.","title":"Flow synthesizer: Universal audio synthesizer control with normalizing flows","authors":"Philippe Esling, Naotake Masuda, Adrien Bardet, Romeo Despres, Axel Chemla-Romeu-Santos","year":"2020","venue":"Applied Sciences","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Garcia, 2002] Ricardo A Garcia. Automatic design of sound synthesis techniques by means of genetic programming. In Audio Engineering Society Convention 113, 2002.","title":"Automatic design of sound synthesis techniques by means of genetic programming","authors":"Ricardo A Garcia","year":"2002","venue":"Audio Engineering Society Convention","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Johnson and Gounaropoulos, 2006] Colin G Johnson and Alex Gounaropoulos. Timbre Interfaces using Adjectives and Adverbs. In Proceedings of the 2006 Conference on New Interfaces for Musical Expression, pages 101-102, 2006.","title":"Timbre Interfaces using Adjectives and Adverbs","authors":"Colin G, Johnson, Alex Gounaropoulos","year":"2006","venue":"Proceedings of the 2006 Conference on New Interfaces for Musical Expression","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Kingma and Welling, 2013] Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. arXiv:1312.6114, 2013.","title":"Auto-encoding variational bayes","authors":"P Diederik, Max Kingma, Welling","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Rezende and Mohamed, 2015] Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International Conference on Machine Learning (ICML), 2015.","title":"Variational inference with normalizing flows","authors":"Mohamed ; Rezende, Danilo Rezende, Shakir Mohamed","year":"2015","venue":"International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Yee-King et al., 2018] Matthew John Yee-King, Leon Fed- den, and Mark d'Inverno. Automatic programming of vst sound synthesizers using deep networks and other tech- niques. IEEE Transactions on ETCI, 2(2), 2018.","title":"Automatic programming of vst sound synthesizers using deep networks and other techniques","authors":"Matthew John, Yee-King, Leon Fedden, Mark D' Inverno","year":"2018","venue":"IEEE Transactions on ETCI","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}