Acquisition of Human Feelings in Music Arrangements

1 Introduction

KANSEI (Japanese; lit. human feelings) [Tsuji, 1995] has been analyzed using quantitative psychological anal­ ysis methods, such as the semantical differential (SD) method and multivariate analysis, which analyze human feelings . Such analyses can isolate feelings associated with known objects, but cannot predict feelings for a new object, nor create a new object for the purpose of generating in the subject a specific feeling. We would like to introduce and discuss a system capable of pre­ dicting feelings and creating new objects based on seed structures that have been extracted and are perceived as favorable by the test subject, such as patterns and colors for pictures, or spectrums and their transition for sounds. Until now, such emergent structures have been obtained only through a random and intractable combi­ nation of elements. In this paper, we explain how this difficulty is overcome using machine learning techniques.

·Presently at Matsushita Electric Industrial Co., Ltd., Kadoma, Osaka, 571, JAPAN

As a representative medium, we focus on a MIDI­ based music arrangement system which is applied to au­ tomatic selection or arrangement in an online KARAO­ KE system. This system is used for automatic down­ loading of MIDI data as requested by a user, or can be used more generally for information retrieval or filtering based on emotional data.

2 Melody and Chords

We attempt to extract a musical structure based on melody and chords as shown in Figure 1. In a musical piece, a function - tonic (T), dominant (D), subdomi­ nant (S) or subdominant minor (SDm) - is assigned to each chord. This paper discusses the extraction of two aspects of the structure (i.e., each chord and a sequence of functions) from which the system derives constraints for assigning chords to a melody (supplemented by func­ tions).

3 Mimicking Arrangements

To investigate the feasibility of generating arrangements automatically, the authors constructed a system that mimics human arrangements as shown in Figure 2.

The chord analyzer assigns a function to each chord by parsing chord progression, and translates scores ar­ ranged by corresponding human composers into train­ ing examples. Each example consists of a chord and its corresponding function, from which the system learns (derives) constraints for automatic arrangement. The system is initially provided with a general theory of harmony, which is refined by a learner called a theory refinement system [Tangkitvanich and Shimura, 1992; Numao and Shimura, 1989; Mooney and Ourston, 1994]. Each note in the score has a function, to which the au­ tomatic arranger assigns a chord.

Figure 3 shows an example of such an arrangement. The system can assign chords based on the general the­ ory of harmony, even though they are mediocre. The­ ory refinement introduces some decorations to the chords and refines the techniques used in the training examples, improving the arrangement.

The present authors prepared a set of training and test examples by the same composer. Figure 4 compares the arrangements produced by the initial and refined theo­ ries with those of the human composer, which shows that refinement based on the training examples increases the number of matched chords in the test examples.

Verbal reports from the subjects characterized the ar­ rangements produced by the initial theory as simple, not interesting and flavorless, while those produced using the refined theory are described as refreshing and novel. Learning using multiple scores enhances variety, but the results lack uniformity.

There is more demand today for professional MIDI arrangers than there was several years ago, since dig­ ital synthesizers, electronic pianos, piano players, on­ line KARAOKE systems, etc. have become more popu­ lar. This type of arrangement system will satisfy these needs by learning and rivaling the skills of professional arrangers.

4 Music Recognition Model

The system outlined above only mimics arranged scores and does not consider human feeling, so we have at­ tempted to develop a system for acquiring music recog­ nition model. Such a model is necessary to achieve auto­ matic arrangement based on situation-dependent human feelings. The authors prepared some musical pieces of 8 bars, and then played them for some subjects in order to get their impressions. This was accomplished using the semantic differential (SD) method. The results were Training Example: ! ' . .

An Arrangement composed using the Initial Theory: ! ' . .

An Arrangement composed using the Refined Theory: ! ' . .

utilized by an inductive logic programming (ILP) system [DeRaedt, 1996] capable of deriving a model for music recognition.

4.1 Training Examples

In order to categorize responses by subjects based on the musical pieces they listened to, the authors selected the following 5 pairs of complementary adjectives: bright - dark, clear - unclear, fast - slow, favorable - unfavorable, stable - unstable. A musical piece is evaluated as one of 7 grades for each pair, such that 1 indicates most bright, and 7 indicates most dark in the spectrum presented by a bright-dark pair.

Let evaluation of a piece P for a pair of adjectives A be E{ for a subject (1 :S E{ :S 7). The system generalizes melody by analyzing notes in each bar based on background knowledge described in Section 4.2. For each piece P consisting of 8 bars: Bar1 ,Bar2 , ···,Bars, the system learns to evaluate A from the following 8 training examples:

where Bari is a sequence of (pitch_name, length). An ex­ ample is considered positive if 5 :S E{ :S 7, and negative if 1 :S E{ :S 4. The system also weighs each learned clause according to the values E{ of the examples used.

The system generalizes chord progression by analyzing successive chords. For each piece P with a sequence of n chords: Chord1,Chord2, ···,Chord,., the system learns to evaluate based on the following n- 1 training examples:

where Chorrh is a combination of the following notes and their function:

Root: i, !Iii, ii, !I iii , iii , iv, ilv, v, !I vi, vi !I vii, vii

Third: major, minor, suspended fourth

Fifth: fifth, augmented, diminished

Seventh: none, major sixth, minor seventh, major seventh

Function: T, D, S, SDm

which constructs 12 x 3 x 3 x 4 x 4 = 1728 combinations. The model describes the relationship among two succes­ sive chords and their functions . The high number combi­ nations and relations results in neither neural networks nor decision trees being appropriate as learning tools. Instead, what is required is inductive logic programming that learns not only attributes and propositional descrip­ tions, but also predicates for finding useful relations in obscure combinations. In the experiment, the authors use a learner similar to FOIL [Quinlan, 1990] except that its background knowledge may also be described using Horn clauses.

4.2 Background Knowledge for ILP

The model is constructed based on background knowl­ edge - definitions of predicates that describe melody and chords. Melody is considered to be a sequence of notes that have pitch and duration described by the fol­ lowing predicates:

• Average duration
• Minimum (lowest) pitch
• Maximum (highest) pitch
• Difference between minimum and maximum pitch
• Pitch transition (rising or falling)

Chords are analyzed based on background knowledge as follows:

root_i (Chord) : The root of Chord is i.

root_v(Chord) : The root of Chord is v.

major(Chord) : Chord has the major third.

fifth(Chord) : Chord has the perfect fifth .

seventh(Chord) : Chord has the minot seventh.

tonic(Chord) : The function of Chord is T.

dom (Chord) : The function of Chord is D.

subdom(Chord) : The function of Chord is S.

succ (Chord!, Chord2) : Chord! and Chord2 are successive chords.

Using the background knowledge for each adjective pair, some predicates adjective_pair(Example, Weight) are de­ rived. E.g. bright-dark(Example of Chords, Weight) to detect bright or dark bar is learned as follows:

4.3 Predicting an evaluation

After the system creates a recognition model for a sub­ ject by processing the results of his/her evaluation of various pieces, it predicts the subject's evaluation of sub­ sequent pieces, which are transformed into a set of ex­ amples for analyzing melody and chords. If an adjec­ tive_pair(Example, w1) is satisfied by k1 examples, the evaluation for the adjective pair is calculated by

4.4 Experiments in Recognition

The present authors prepared 100 well-known music pieces, from which they extracted 8 successive bars with­ out modulation. The subject evaluates 85 of the 100 pieces, and the results for adjective pairs are studied by the system to predict evaluations for the other 15 pieces. They then take an average of the results by 8 different subjects.

Let F~ be a prediction of the evaluation of a piece p for adjective pair A, and n be the number of pieces. The difference between the prediction and the evaluation by a subject is shown to be:

Figure 5 shows the average difference for the subjects: ~ l:::~=l diffA, when the prediction is based on chords or melody. Figure 6 shows the percentage of correct pre­ dictions. The results show that the differences are be­ tween 1.0 and 2.0 in the 7-grade ·evaluation, and that on average 70% of predictions are correct, which indicates that the system is capable of predicting evaluation of new pieces by the subjects very well. For criteria fast, slow and unfavorable, the difference increases when the prediction is based on chords. This means that these im­ pressions are based mainly on melody and particularly from the length of each note. The system predicts re­ sponses within the criteria bright and dark from chords very well. These impressions are dependent mainly on whether the third is major or minor. Although the sys­ tem cannot discern these impressions in melody, the au­ thors are now attempting to detect them by analyzing sequences of notes.

Figure 7 shows the variance of impressions among the subjects, which is relatively large in the adjective pairs fast-slow and stable-unstable. These subject-dependent pairs are learned by the system very well according to Figures 5 and 6.

This level of arrangement will be employed in KARA­ OKE systems for generating a user model of emo­ tional data to extract an appropriate piece from a MIDI database, or to select suitable background pictures or video for each title.

5 Arrangement based on the Model

Utilizing the music recognition model, the system con­ trols the arrangement process based on its overall mood. To improve the arrangement, we assume that a human arranger ( or the system shown in section 3 ) composes the original arrangement, which is then modified slightly to result in a specified impression.

5.1 Modifying a Score to Change its Impression

Before arrangement, the system develops a music recog­ nition model for the user. Information about the mood of the user is provided in the form of a 7-grade scale for each of the adjective pairs. The chord progression of a score is modified according to the following priorities:

1. Remove any chord or chord progression whose im­ pression is described by the opposite adjective in the pairs.
2. To minimize differences from the original score, avoid:
3. (a) modifying any chord and chord progression that satisfy the given adjective,
4. (b) changing any functions, and
5. (c) modifying any chord and chord progression that is not in opposition to the given adjective.
6. Modify chords to minimize differences in the evalu­ ation.

5.2 Experiments in Arrangement

The authors prepared 94 well-known music pieces with­ out modulation, from which they extracted 4 or 8 suecessive bars. The subject evaluates 80 of the 94 pieces in 3 pairs of adjectives: bright - dark, favorable - unfavor­ able, stable-unstable, and the results of the evaluations are processed by the system to modify the chord pro­ gressions of 6 of the 14 pieces in 6 criteria - bright, dark, favorable, unfavorable, stable and unstable. The subject evaluates the modified 6 x 6 = 36 pieces and the original 6 pieces without being notified of how each piece has been modified.

The present authors repeated the above experiment using 8 subjects. Table 1 shows the percentage of ar­ rangements whose modifications corresponded to the in­ tended changes, as well as their standard deviation. Ac­ cording to the table, intended arrangements are pro­ duced on average 60% of the time. Table 2 shows the percentage of arrangements for which the subjects' im­ pression is improved and the standard deviation.

Figure 8 shows the average percentage of pieces for which the impression is improved by the modification. According to the figure, 60% of arrangements are im­ proved for the criteria brightness or darkness. In general, if an arranger is less than proficient, the results tend to be evaluated as unstable or unfavorable. Therefore, it is more likely that a piece will be arranged and evaluated as unstable or unfavorable than as stable or favorable .

According to the figure, some arrangements are evalu­ ated as stable or favorable, although these make up less than half of the total. This suggests that an arranger based on learning personal feelings is effective.

If an original arrangement was evaluated as very bright, it would be difficult to rearrange the piece for a higher level of brightness. It would be easier to rear­ range the piece to less bright. Figure 9 shows the average percentage of pieces successfully rearranged to cause the opposite impression. According to the figure, 80% of arrangements are rearranged in this fashion.

6 Related Works

Widmer [1994] proposed a method of accomplishing explanation-based learning by attaching harmonies - chord symbols to the notes of a melody. This task is closely related to that described in Section 3. The present paper further discusses a means of controlling the process based on learned feelings.

Katayose, Imai and Inokuchi [1988] approached the understanding of music based on the following rules:

Melody:

Both the fourth and the seventh do not exist

-+ Oriental mood

Our system not only uses this kind of rule but also cre­ ates it and applies it through a process that considers its weight.

Most applications of machine learning related to music investigate approaches to interpretation of a music score for playing, for example, measuring the loudness of each note in a score [Widmer, 1993; 1994], or acquiring play­ ing viola [Furukawa, 1997; 1996].

Neural Networks have been used for dealings with problems in music [Todd and Loy, 1991]. The present authors believe that inductive logic programming offers a better means of describing music scores, in which struc­ tures are very important. To quantify feelings in music, we introduce a weight W1 to each clause. A better so­ lution is to combine logic programming and neural net­ works by weie;hting links in dynamic logical networks, as described in [Numao et al., 1997].

7 Conclusion

We present an approach for utilizing human feeling in listening to and composing music. This approach to feel­ ings is an interesting test of the application of machine learning techniques, and should be developed further due to the lack of practical tests to date. If background knowledge of the learner incorporates some important theories in musicology and psychology [Meyer, 1956; Hiraga, 1987; Longuet-Higgins, 1987], we may obtain a more powerful tool for composing music and analyzing human feelings.

Acknowledgments

An earlier version of the system· was implemented by Takashi Shirai (Nintendo), Koji Yamaguchi (Matsushita Electric) and Masatake Saito (SONY).