Pattern-Based Music Generation with Wasserstein Autoencoders and PRC Descriptions

1 Introduction

During the last decade, the availability of new and powerful methods based on deep learning has sparkled a flourishing line of research in which generative models trained on data have been applied to the generation of music in various modalities, styles, and genres [Briot et al., 2019]. In spite of significant recent advancements in this area, results are still not always usable in realistic music scenarios such as studio production using standard Digital Audio Workstations (DAWs) and live performance of electronic music. A detailed review is impossible here due to space limitations, but at a high level we can identify two main streams of research for MIDI music generation. The first one describes MIDI partitures as temporal sequences of events, suitable to be handled by generative models based on various types of recurrent networks [Boulanger-Lewandowski et al., 2012; Huang et al., 2019; Roberts et al., 2018]. Generation of multiinstrument polyphonic music in this setting is however difficult. The second one, used in this work, focuses on patterns that consist of a fixed amount of bars (e.g., two or four), and that are described as pianorolls (PR) [Yang et al., 2017; Dong et al., 2018; Dong and Yang, 2018].

We introduce some important variants compared to previous works, whose combination allows us to generate meaningful and professionally usable streams of music. These include: (1) the use of a novel PR-like pattern description, what we call PR C1 , (2) the use of Wasserstein autoencoders

∗Contact Author

1 The C after Conlon Nancarrow (1912-1997), a pioneer of piano

(WAE) [Tolstikhin et al., 2018] as generative learners; and (3) the definition of an optimization strategy for exploring the autoencoder latent space during generation.

2 Materials andMethods

2.1 Datasets

We assembled two datasets of 910 and 968 patterns respectively, especially composed by two professional musicians for this study. In both cases we instructed composers to create coherent 120bpm patterns of four bars that were quantized at 1/32 resolution and limiting the extension to four octaves, yielding tensors of size T = 128 × N = 48 × 2 for each track. Dataset A contains patterns in the genres of acid jazz, soul and funk with four instruments (drums, bass, Rhodes piano, and Hammond organ). Dataset B contains patterns in the high-pop and progressive trance genres and has ten instruments (guitar, two pads, strings, choir, and lead, in addition to the above four). All patterns were transposed to prevent tonality variations.

2.2 MIDI Patterns in PR C

Each track in a MIDI stream consists of a sequence of time stamped events. We consider here only two types of note events: ON(t, n, v), and OFF(t, n) where t denotes the time at which note n begins or ends and the note identifier, n, takes values in general from 0 (C-1) to 127 (G9) but only 4 octaves were used in this work. The value, v, is the MIDI velocity in the integer range [0, 127]. Roughly, velocity is associated with the note intensity (allowing to represent dynamic expression elements, such as pianissimo, forte, or accents in percussive instruments) but depending on the synth attached to the track, it can also affect timbre. After time quantization is applied, a pattern consisting of a fixed number of bars can be described as a tensor, a strategy previously proposed in [Dong et al., 2018] but in our PR C description we employ a T ×N×2 tensor x, where T is the number of legal time positions, N the number of notes. In the first channel, xq(t),n,1 = v if there occurs a note-on event ON(t, n, v) and zero otherwise. In the second channel, xq(t),n,2 = d if there occurs a note-on event ON(t, n, v) and zero otherwise, but in this case d is the duration (expressed in quantized steps) of the note. Note that

roll compositions. Here Conlon also stands for Channeled Onset of Notes and Length Of Notes.

polyphony is handled naturally in this description and multiinstrument patterns with K tracks can be easily described by allocating two channels for each track as above, resulting in a T ×N × 2K tensor. Unlike the PR used in [Dong et al., 2018], our PR C description does not suffer the ambiguity between long notes and repeated occurrences of the same note and is completely lossless (a quantized MIDI pattern transformed into the corresponding PR C tensor can be recovered exactly). Additionally, the PR in [Dong et al., 2018] consists of Boolean tensors (after applying a hard threshold to velocity), which discard dynamic expressiveness.

2.3 Wasserstein Autoencoders

Generative models that have been applied to music include variational autoencoders (VAE) [Kingma and Welling, 2014] and generative adversarial networks (GAN) [Goodfellow et al., 2014]. In both cases, a network (called either decoder or generator) is trained to map a latent or noise vector z ∈ RD into a pattern. Here we are interested in autoencoder based approaches (see also below for a motivation). The regularizer in VAEs is well known to originate a blurriness phenomenon in the case of images [Dosovitskiy and Brox, 2016]. In the case of music, we observed that "blurriness" also occurs, resulting in large clusters of notes being played together and in sometimes obsessive repetitions of short notes. WAEs [Tolstikhin et al., 2018] avoid this problem by penalizing a measure of discrepancy between the expected p(z|x) and the prior p(z), i.e., by pushing the expectation inside the distance. We employed the maximum mean discrepancy measure and a Gaussian prior. Encoder and decoder were structured as in the DCGAN [Radford et al., 2016] architecture with large filter sizes (8 × 8) and 256 filters in the middle layers. A very small latent vector size (D = 3) was sufficient to ensure good reconstruction of training patterns. We used the square loss (with tanh output units) to measure the reconstruction error.

2.4 Generation of Pseudo-Songs

We define two modalities for producing pseudo-songs: interpolation and swirls. In the first case (only applicable to autoencoders), users have the choice of picking a start pattern xs and a goal pattern xg (both from the test set) and the system creates a smooth interpolation between the two. In particular, embeddings z1 = E(xs) and zT = E(xg) are first created using the encoder E, and then a spherical [White, 2016] trajectory in the embedding space is computed as zt = (sin( 1-t θ(1-T ) )z1+sin(θ t-T 1-T )zT )/ sin(θ) where θ = arccos(z1/‖z1‖, zT/‖zT‖). In the case of swirls (also applicable to GANs), latent trajectories are produced deterministically by taking real and imaginary parts of periodic complex-valued parametric functions of the form

Note that equally spaced points in the embedding space do not necessarily correspond to equally spaced reconstructions in the pattern space. In some cases, this may lead to abrupt transitions, and in some other cases to "flat" progressions that might be musically uninteresting. To address this issue we subsampled the trajectory by maximizing the minimum Euclidean distance between consecutive reconstructions, a problem that has a standard solution in terms of the bottleneck path problem [Kaibel and Peinhardt, 2006] in a Viterbi-like T × L trellis (arcs weighted by distance), being L < T the desired duration of the pseudo-song. The problem is solvable by a simple modification of the Dijkstra algorithm.

3 Results and Discussion

Generated pseudo-songs with PR C are consistent over time in terms of harmony, rhythmic and melodic flow and style, with accurate interplay between the instruments. Discontinuities and choppy results often occur when using the PR description. Results with PR C are consistently better than those obtained with PR also quantitatively, as measured by considering test set reconstructions. In Table 1 we report precision and recall (considering as predictions notes in the reconstruction whose predicted velocity is above the minimum velocity in the training data) and, in the true positives set, the mean absolute error for the velocity (in the range [0 - 127]) and for the duration (in units of 1/32ths of bar). For PR the predicted note-on event was the first element in the merged row of consecutive predictions.

To allow musicians to use pseudo-song generation in production and performance settings, an API for accessing trained networks over the internet was developed. It takes a specification in the form of the start and goal pattern and the required length, L, and returns a MIDI file for the interpolation. Specification by the user and delivery is arranged through a purpose-built web client or a plug-in for DAWs. LiveAI RP (https://www.musi-co.com/listen/live/research) is a web client that lets the user generate interpolations from WAEs trained on datasets A and B. The resulting MIDI files can be downloaded for production use in any DAW.

Earlier experiments have shown that musicians appreciate interpolation as an understandable and useful form of music generation [Borghuis et al., 2018]. Interpolations can sound like directed musical flows, as the music "leaves the start pattern behind" and the "attraction" exerted by the goal manifests itself by elements of the goal pattern entering the music. However, the coherence of pseudo-songs is entirely due to the properties of the embedding space: points in close proximity have reconstructions in pattern space that are musically close. Thus, a natural direction for future development is to incorporate mechanisms of conditioning by labeling of dataset patterns with structural categories (e.g. intro, verse, chorus).