Musical Composition Style Transfer via Disentangled Timbre Representations

1 Introduction

Music generation has long been considered as an important task for AI, possibly due to the complicated mental and creative process that is involved, and its wide applications in our daily lives. Even if machines cannot reach the same professional level as well-trained composers or songwriters, machines can cooperate with human to make music generation easier or more accessible to everyone.

Along with the rapid development of deep generative models such as generative adversarial networks (GANs) [Goodfellow and others, 2014], the research on music generation is experiencing a new momentum. While a musical work is typically composed of melody, chord, and rhythm (beats), a major trend of recent research focuses on creating original content for all or some of these three parts [Roberts et al., 2017;

Dong et al., 2018a]. That is, we expect machines can learn the complex musical structure and compose music like human composers. This has been technically referred to as music composition [Briot et al., 2017].

Another trend is to generate variations of existing musical material. By preserving the pitch content of a musical piece, we expect machines to modify style-related attributes of music, such as genre and instrumentation. This has been technically referred to as music style transfer [Dai et al., 2018]. A famous example of style transfer, though not fully-automatic, is the "Ode to Joy" project presented by [Pachet, 2016]. The project aims to rearrange (or reorchestrate) Beethoven's Ode to Joy according to the stylistic conventions of seven other different musical genres such as Jazz and Bossa Nova. 1

In this paper, we are in particular interested in such a music rearrangement task. The work of [Pachet, 2016] is inspiring, but as the target styles are by nature very different, they had to tailor different machine learning models, one for each style. Moreover, they chose to use non-neural network based models such as Markov models, possibly due to the need to inject musical knowledge manually to ensure the final production quality. As a result, the solution presented by [Pachet, 2016] is not general and it is not easy to make extensions. In contrast, we would like to investigate a universal neural network model with which we can easily change the instrumentation of a given music piece fully automatically, without much affecting its pitch content. While we do not think such a styleagnostic approach can work well for all the possible musical styles, it can be treated as a starting point from which stylespecific elements can be added later on.

Music rearrangement is closely related to the art of music arrangement [Corozine, 2015]. For creating music, it involves not only composing the different parts (e.g., melody, chords) of a musical work but also arranging/selecting the instruments to play the different parts. For example, chord and melody can be played by different instruments. Setting the instrumentation is like coloring the music. Therefore, music arrangement is an "advanced" topic, and is needed towards generating fully-fledged music material. Compared to music arrangement, music rearrangement is a style transfer task and is like re-coloring an existing music piece. By contributing to music rearrangement, we also contribute indirectly to the more challenging automatic music arrangement task, which has been seldom addressed in the literature as well.

1 [Online] https://www.youtube.com/watch?v=buXqNqBFd6E

Music rearrangement is not an easy task even for human beings and requires years of professional training. In order to convincingly perform music style transfer between two specific styles, a musician must first be familiar with the annotation, instrument usage, tonality, harmonization and theories of the two styles. The task gets harder when we consider more styles. For machines to rearrange a musical piece, we have to consider not only the pitch content of the given music, but also the pitch range of each individual instrument as well as the relation between different instruments (e.g., some instrument combination creates more harmonic sounds). And, it is hard to find paired data demonstrating different instrumentations of the same music pieces [Crestel et al., 2017]. A neural network may not learn well when the data size is small.

To address these challenges, we propose to train music transcription neural networks that take as input an audio file and generate as output the corresponding musical score. In addition to predicting which notes are played in the given audio file, the model also has to predict the instrument that plays each note. As a result, the model learns both pitch- and timbre-related representations from the musical audio. While the model is not trained to performmusic rearrangement, we investigate "disentanglement techniques" such as adversarial training [Liu et al., 2018b; Lee et al., 2018] to separate latent factors of the model that are pitch- and timbre-related. By disentangling pitch and timbre, the model has an idea of how a musical piece was composed and arranged. If the disentanglement is well done, we expect that we can rearrange a musical piece by holding its pitch-related latent factors fixed while manipulating the timbre-related ones.

We propose and investigate two models in this paper. They share the following two advantages. First, they require only pairs of audio and MIDI files, which are relatively easier to collect than different arrangements of the same music pieces. Second, they can take any musical piece as input and rearrange it, as long as we have its audio recording.

In sum, the main contributions of this work include:

• To the best of our knowledge, this work presents the first deep learning models that realize music rearrangement, a composition style transfer task [Dai et al., 2018] (see Section 2.1), for polyphonic music of arbitrary genres.
• While visual attribute disentanglement has been much studied recently, attribute disentanglement for musical audio has only been studied in our prior work [Hung et al., 2018], to our knowledge. Extending from that work, we report comprehensive evaluation of the models, emphasizing their application to rearrangement.

Musical compositional style transfer is still at its early stage. We hope this work can call for more attention toward approaching this task with computational means.

2 RelatedWork

2.1 Music Style Transfer

According to the categorization of [Dai et al., 2018], there are three types of music style transfer: timbre style transfer, performance style transfer, and composition style transfer. To our knowledge, timbre style transfer receives more attention than the other two lately. The task is concerned with altering the timbre of a musical piece in the audio domain. For example, [Engel et al., 2017] applied WaveNet-based autoencoders to audio waveforms to model the musical timbre of short single notes, and [Lu et al., 2019] employed a GANbased network on spectrograms to achieve long-structure timbre style transfer. However, for the latter two types of music style transfer, little effort has been made.

Music rearrangement can be considered as a composition style transfer task, which is defined as a task that "preserve[s] the identifiable melody contour (and the underlying structural functions of harmony) while altering some other score features in a meaningful way" [Dai et al., 2018]. The recent work on composition style transfer we are aware of are that by [Pati, 2018] and [Kaliakatsos-Papakostas et al., 2017], which used neural networks and rules respectively to deal with melody, rhythm, and chord modification. Our work is different in that we aim to modify the instrumentation.

2.2 Disentanglement for Style Transfer

Neural style transfer was first introduced in image style transfer [Gatys et al., 2016]. In this work, they proposed to learn an image representation to separate and recombine content and style. Since then, several methods have been proposed to modify the style of a image. [Liu et al., 2018b] trained an auto-encoder model supervised by content ground truth labels to learn a content-invariant representation, which can be modified to reconstruct a new image. [Lee et al., 2018] showed that it is possible to learn a disentangled representation from unpaired data by using cross-cycle consistency loss and adversarial training, without needing the ground truth labels.

Disentangled representation learning for music is still new and has only been studied by [Brunner et al., 2018] for symbolic data. By learning a style representation through style classifier, their model is able to modify the pitch, instrumentation and velocity given different style codes. Our model is different in that we deal with audio input rather than MIDIs.

3 ProposedModels

Figure 1 shows the architecture of the proposed models. We opt for using the encoder/decoder structures as the backbone of our models, as such networks learn latent representations (a.k.a., latent codes) of the input data. With some labeled data, it is possible to train an encoder/decoder network in a way that different parts of the latent code correspond to different data attributes, such as pitch and timbre. Both models use encoders/decoders and adversarial training to learn music representations, but the second model additionally uses skip connections to deal with the pitch information. We note that both models perform polyphonic music transcription. We present the details of these models below.

3.1 Input/Output Data Representation

The input to our models is an audio waveform with arbitrary length. To facilitate pitch and timbre analysis, we firstly convert the waveform into a time-frequency representation that shows the energy distribution across different frequency bins for each short-time frame. Instead of using the short-time Fourier transform, we use the constant-Q transform (CQT), for it adopts a logarithmic frequency scale that better aligns with our perception of pitch [Bittner et al., 2017]. CQT also provides better frequency resolution in the low-frequency part, which helps detect the fundamental frequencies.

Figure 2: Different symbolic representations of music.

As will be described in Section 3.5, our encoders and decoders are designed to be fully-convolutional [Oquab et al., 2015; Liu et al., 2019], so that our models can deal with input of any length in testing time. However, for the convenience of training the models with mini-batches, in the training stage we divide the waveforms in our training set into 10-second chunks (without overlaps) and use these chunks as the model input, leading to a matrix Xcqt ∈ RF×T of fixed size for each input. In our implementation, we compute CQT with the librosa library [McFee et al., 2015], with 16,000 Hz sampling rate and 512-sample window size, again with no overlaps. We use a frequency scale of 88 bins, with 12 bins per octave to represent each note. Hence, F = 88 (bins) and T = 312 (frames).

We use the pianorolls [Dong et al., 2018b] (see Figure 2a) as the target output of our models. A pianoroll is a binaryvalued tensor that records the presence of notes (88 notes here) across time for each track (i.e., instrument). When we consider M instruments, the target model output would be Xroll ∈ {0, 1}F×T×M . Xroll and Xcqt are temporally aligned, since we use MIDIs that are time-aligned with the audio clips to derive the pianorolls, as discussed in Section 3.5. As shown in Figures 1 and 2, besides asking our models to generate Xroll, we use the instrument roll Xt ∈ {0, 1}M×T and pitch roll Xp ∈ {0, 1}F×T as supervisory signals to help learn the timbre representation. Both Xt and Xp can be computed from Xroll by summing along a certain dimension.

3.2 The DuoEDModel

The architecture of DuoED is illustrated in Figure 1a. The design is inspired by [Liu et al., 2018b], but we adapt the model to encode music. Specifically, we train two encoders Et and Ep to respectively convert Xcqt into the timbre code Zt = Et(Xcqt) ∈ Rκ×τ and pitch code Zp = Ep(Xcqt) ∈ Rκ×τ . We note that, unlike in the case of image representation learning, here the latent codes are matrices, and we require that the second dimensions (i.e., τ) represent time. This way, each column of Zt and Zp is a κ-dimensional representation of a temporal segment of the input. For abstraction, we require κτ < FT .

DuoED also contains three decoders Droll, Dt and Dp. The encoders and decoders are trained such that we can use Droll([ZTt , ZTp ]T ) to predict Xroll, Dt(Zt) to predict Xt, and Dp(Zp) to predict Xp. The prediction error is measured by the binary cross entropy between the ground truth and the predicted one. For example, for the timbre classifier Dt, it is:

where ̂Xt = Dt(Zt), '·' denotes the element-wise product, and σ is the sigmoid function that scales its input to [0, 1]. We can similarly define Lroll and Lp.

In each training epoch, we optimize both the encoders and decoders by minimizing Lroll, Lt and Lp for the given training batch. We refer to the way we train the model as using the "temporal supervision," since to minimize the loss terms Lroll, Lt and Lp, we have to make accurate prediction for each of the T time frames.

When the adversarial training strategy is employed (i.e., those marked by dashed lines in Figure 1a), we additionally consider the following two loss terms:

where ̂Xnt = Dt(Zp), ̂Xnp = Dp(Zt), meaning that we feed the 'wrong' input (purposefully) to Dt and Dp.

The equation must close to zero, which means when we use the wrong input, we expect Dt and Dp can output nothing (i.e., all zeros), since, e.g., Zt is supposed not to contain any pitch-related information.

Please note that, in adversarial training, we use Lnt and Lnp to update the encoders only. This is to preserve the function of the decoders in making accurate predictions.

3.3 The UnetEDModel

The architecture of UnetED is depicted in Figure 1b, which has a U-shape similar to [Ronneberger et al., 2015]. In UnetED, we learn only one encoder Ecqt to get a single latent representation Zt of the input Xcqt. We add skip connections between Ecqt and Droll, which enables lower-layer information of E (closer to the input) to be passed directly to the higher-layer of D (closer to the output), making it easier to train deeper models. The model learn Ecqt and Droll by minimizing Lroll, the cross entropy between Droll(Zt) and the pianoroll Xroll. Moreover, we promote timbre information in Zt by refining Ecqt and learning a classifier Dt by minimizing Lt (see Eq. (1)). When the adversarial training strategy is adopted, we design a GAN-like structure (i.e., unlike the case in DuoED) to dispel pitch information from timbre representation. That is, the model additionally learns a pitch classifier Dp to minimize the loss Lp between Dp(Zt) and Xp. This loss function only updates Dp. Meanwhile, the encoder Ecqt tries to fool Dp, where the ground truth matrices should be zero matrices. That is, the encoder should minimize Lnp , and the loss function only affects the encoder.

The design of UnetED is based on the following intuitions. First, since Zt is supposed not to carry pitch information, the only way to obtain the pitch information needed to predict Xroll is from the skip connections. Second, it is fine to assume that the skip connections can pass over the pitch information, due to the nice one-to-one time-frequency correspondence between Xcqt and each frontal slice of Xroll. 2 Moreover, in Xcqt pitch only affects the lowest partial of a harmonic series created by a musical note, while timbre affects all the partials. If we view pitch as the "boundary" outlining an object, and timbre as the "texture" of that object, detecting the pitches may be analogous to image segmentation, for which U-nets have been shown effective in solving [Ronneberger et al., 2015].

We note that the major difference between DuoED and UnetED is that there is a pitch code Zp in DuoED. Although the pitch representation may benefit other tasks, for composition style transfer we care more about the timbre code Zt.

3.4 Composition Style Transfer

While both DuoED and UnetED are not trained to perform music rearrangement, they can be applied to music rearrangement due to the built-in pitch/timbre disentanglement. Specifically, for style transfer, we are given a source clip A and a target clip B, both audio files. We can realize style transfer by using A as the input to the encoder of DuoED or UnetED to get content information (i.e., through the pitch code Z(A) p or the skip connections), but then combine it with the timbre code Z(B) t obtained from B to generate an original pianoroll X(A)→(B) roll , from which we can create synthesized audio.

3.5 Implementation Details

Since the input and output are both matrices, we use convolutional layers in all the encoders and decoders of DuoED and UnetED. To accommodate input of variable length, we adopt a fully-convolutional design [Oquab et al., 2015], meaning that we do not use pooling layers at all. All the encoders in our models are composed of four residual blocks. Each block contains three convolution layers and two batch normalization layers. The decoder Droll has the same structure, but use transpose convolution layers to do the upsampling. There are in total twelve layers in both encoder and pianoroll decoder Droll. For the pitch and timbre decoder, we use three transpose convolution layers to reconstruct the pitch roll and instrument roll from latent representation. Moreover, we use leaky ReLU as the activation function for all layers but the last one, where we use the sigmoid function. Both DuoED and UnetED are trained using stochastic gradient descend with momentum 0.9. The initial learning rate is set to 0.005.

2 That is, both Xcqt(i, j) and Xroll(i, j,m) refer to the activity of the same musical note i for the same time frame j.

We use the newest released MuseScore dataset [Hung et al., 2019] to train the proposed models. This dataset contains 344,166 paired MIDI and MP3 files. Most MP3 files were synthesized from the MIDIs with the MuseScore synthesizer by the uploaders. Hence, the audio and MIDIs are already time-aligned. We further ensure temporal alignment by using the method proposed by [Raffel and Ellis, 2016]. We then convert the time-aligned MIDIs to pianorolls with the pypianoroll package [Dong et al., 2018b]. The dataset contains music of different genres and 128 different instrument categories as defined by the MIDI spec.

4 Experiment

As automatic music rearrangement remains a new task, there are no standard metrics to evaluate our models. We propose to firstly evaluate our models objectively with the surrogate task of instrument activity detection (IAD) [Gururani and Lerch, 2017]-i.e., detecting the activity of different instruments for each short-time audio segment-to validate the effectiveness of the obtained instrument codes Zt. After that, we evaluate the performance of music rearrangement subjectively, by means of a user study.

4.1 Evaluation on Instrument Activity Detection

We evaluate the performance for IAD using the 'MedleyDB + Mixing Secret' (M&M) dataset proposed by [Gururani and Lerch, 2017]. The dataset contains real-world music recordings (i.e., not synthesized ones) of various genres. By evaluating our model on this dataset, we can compare the result with quite a few recent work on IAD. Following the setup of [Hung et al., 2019], we evaluate the result for per-second IAD in terms of the area under the curve (AUC). We compute AUC for each instrument and report the per-instrument AUC as well as the average AUC across the instruments.

IAD is essentially concerned with the prediction of the instrument roll shown in Figure 2b. As our DuoED and UnetED models are pre-trained on a synthetic dataset MuseScore, we train additional instrument classifiers D′t with the pre-defined training split of the M&M dataset (200 songs) [Hung et al., 2019]. D′t takes as input the timbre code Zt computed by the pre-trained models. Following [Gururani et al., 2018], we use a network of four convolution layers and two fully-connected layers for the instrument classifiers D′t . 3 We use the estimate of D′t as the predicted instrument roll.

3 We do not use the M&M training split to fine-tune the original

Table 1 shows the evaluation result on the pre-defined test split of the M&M dataset (69 songs) [Hung et al., 2019] of four state-of-the-art models (i.e., the first four rows) and our models (the middle two rows), considering only the five most popular instruments as [Hung et al., 2019]. Table 1 shows that the proposed UnetED model can achieve better AUC scores in most instruments and achieve the highest average AUC 0.829 overall, while the performance of DuoED is on par with [Gururani et al., 2018]. We also conduct a paired t-test and found that the performance difference between Gururani's method and the proposed UnetED method is statistically significant (p-value<0.05). This result demonstrates the effectiveness of the learned disentangled timbre representation Zt for IAD, compared to conventional representations such as the log mel-spectrogram used by [Gururani et al., 2018]. Moreover, we note that these prior arts on IAD cannot work for composition style transfer, while our models can.

The last two rows of Table 1 show the result of two existing disentanglement methods originally developed for images [Liu et al., 2018a; Hadad et al., 2018]. 4 We use encoder/decoder structures similar to our models and the same strategy to pre-train on MuseScore and then optimize the timbre classifier on the M&M training split. We see that the proposes models still outperform these two models. This can be expected as their models were not designed for music.

instrument classifiers Dt of our models but train a new one D′t, so as to make sure that our instrument classifier has the same architecture as that used by [Gururani et al., 2018]. In this way, the major difference between our models and theirs is the input feature (the timbre code Zt for ours, and the log mel-spectrogram for theirs), despite of some minor differences in for example the employed filter sizes.

4 We adapt the two methods for music as follows. We employ the

Table 2 reports an ablation study where we do not use the M&M training split to learn D′t but use the original Dt trained on MuseScore for IAD. In addition, we compare the case with or without adversarial training, and the case where we get the instrument roll estimate from the output of Droll [Hung et al., 2019], which considers both pitch and timbre. We see that adversarial training improves IAD, and that adding pitch information helps. From Tables 1 and 2 we also see that it is helpful to update the instrument classifier by the training split of M&M, possibly due to the acoustic difference between synthetic and real-world audio recordings.

4.2 Evaluation onMusic Rearrangement

To evaluate the result of music rearrangement, we choose four types of popular composition style in our experiment: strings composition (i.e., violin and cello), piano, acoustic composition (i.e., guitar, or guitar and melody), and band composition (i.e., electric guitar, bass, and melody); the melody can be played by any instrument. We randomly choose four clips in each type from the MuseScore dataset, and then transfer them into the other three types following the approach described in Section 3.4. We intend to compare the result of UnetDE, DuoDE, UnetDE w/o Adv, and the method proposed by [Hadad et al., 2018]. We treat the last method as the baseline, as it is designed for image style transfer, not for music.

pianoroll as the target output for both. And, for [Liu et al., 2018a], we treat 'S' as timbre and 'Z' as pitch; for [Hadad et al., 2018], we replace 'style' with pitch and 'class' with timbre.

We invite human subjects to listen to and rate the rearrangement result. Each time, a subject is given the source clip, and the four rearranged results by different models, all aiming to convert the style of that clip to one of the other three. The subject rate the rearranged results in the following three dimensions in a four-point Likert scale:

• whether the composition sounds rhythmic;
• whether the composition sounds harmonic;
• and, the overall quality of the rearrangement.

The scores are the higher the better. Since there is no ground truth of music style transfer, the rating task is by nature subjective. This process is repeated 4×3 = 12 times until all the source-target pairs are evaluated. Each time, the ordering of the result of the four modes are random.

We distributed the online survey through emails and social media to solicit voluntary, non-paid participation. 40 subjects participated in the study, 18 of which have experience in composing music. Figure 3 shows the average result across the clips and the subjects. It shows that, while DuoED and UnetED perform similarly for IAD, UnetED performs better in all the three metrics for music rearrangement. Moreover, with the adversarial training, the UnetED can generate more rhythmic and harmonic music than its ablated version. And, [Hadad et al., 2018] does not work well at all. 5 Moreover, we found that there is no much difference between the ratings from people with music background and people without music background. The only difference is that people with music background averagely tends to give slightly lower ratings.

Figures 4 shows examples of the rearrangement result by UnetED. We can see that when rearranging the music to be played by band, UnetED finds the low-pitched notes for the bass to play and the melody notes for the flute or violin to play. This demonstrates that UnetED has the ability to choose suitable instruments playing certain notes combination. This result is also showed in arranging string instruments. More demo results of UnetED can be found at https://biboamy.github.io/disentangle demo/.

5 The poor result of [Hadad et al., 2018] can be expected, since the pre-trained instrument decoder can only guarantee Zt to contain timbre information, but cannot restrict Zt not to have any pitch information. We observe that this method tend to lose much pitch information in the rearranged result and create a lot of empty tracks.

5 Conclusion

In this paper, we have presented two models for learning disentangled timbre representations. This is done by using the instrument and pitch labels from MIDI files. Adversarial training is also employed to disentangle the timbre-related information from the pitch-related one. Experiment show that the learned timbre representations lead to state-of-the-art accuracy in instrument activity detection. Furthermore, by modifying the timbre representations, we can generate new score rearrangement from audio input.

In the future, we plan to extend the instrument categories and also include the singing voice to cope with more music genres. We want to test other combinations of instruments to evaluate the performance of our models. We also want to further improve our models by re-synthesizing the MP3 files of the MuseScore data with more realistic sound fonts.