Automatic Decision of Piano Fingering Based on a Hidden Markov Models

1 Introduction

This paper proposes an algorithm for automatic decision of piano fingering based on Hidden Markov Models (HMMs, see [Huang et al., 2001]). The goal is to formulate the reasonable hand motion in a piano performance as well as achieving highly reliable decision of piano fingering. This algorithm is expected to be utilized for presenting a model performance to piano learners, estimating the required playing skill of piano pieces, etc. If considered as part of the motion planning of a robot piano player, the proposed approach in this paper may be widely applicable to more general robot's manipulation tasks using robot fingers.

A number of existing papers on piano fingering decision dealt with monophonic melodies in a single hand through statistical methods [Noguchi et al., 1996], fingering cost functions defined in between two notes [Hart et al., 2000; Kasimi et al., 2005], or rule-based methods [Hayashida and Mizutani, 2003]. In [Kasimi et al., 2005], chords and polyphonic melodies in a single hand were also studied. However, the statistical method presented in [Noguchi et al., 1996] required score data tagged with fingering information and the rule-based approach [Hayashida and Mizutani, 2003] needed to handle conflict resolution between applicable rules.

Past research on automatic fingering decision seemed to focus on the score-to-fingering conversion. Instead, we focus on another aspect of fingering-to-performance conversion which can be easily formulated mathematically by introducing a probabilistic approach.

2 Automatic Decision of Piano Fingering

2.1 Formulation of Piano Fingering

Automatic decision of piano fingering can be formulated as a problem to find the most likely fingering adopted by welltrained piano players. It essentially involves probabilistic modeling of piano performance.

The process by which a piano performer plays a sequence of notes given in a score or in MIDI format can be considered as consisting of the following three steps:

1. read the input "score",
2. decide the "fingering",
3. and produce the music sounds ("performance").

Direct score-to-fingering conversion (from (1) to (2)) may face a difficulty in finding a reasonable local objective function representing the ease of playing. For example, the sum of the fingering costs in [Hart et al. et al., 2005] lacks mathematical and well-defined meanings.

In contrast, we focus on another natural order, i.e., the fingering-to-performance conversion (from (2) to (3)). We will design a fingering model where we can incorporate various probabilistic factors representing physical constraints of fingering and can design a local objective function mathematically. Considering an imaginary piano performance, we can determine the most likely fingering for producing that imaginary performance through the maximum a posteriori (MAP) approach. If the imaginary performance perfectly corresponds to the given score, the over-all function becomes automatic fingering decision for the given score.

2.2 Mathematical Model of Fingering with HMM

Fingering would be no problem unless music notes were preceded and/or followed by other notes. Fingering is actually a problem because one note (or more notes) should be played after another (or others) within the time of the note length assigned to the preceding notes. This implies that fingering is essentially a problem of finding an optimal sequence of smooth state transition, typically, from a state of one finger pressing a key (corresponding a note pitch) to another state of another finger pressing another key. Thus, piano performance can be interpreted as the process of generating a sequence N of performed notes from a sequence S of states of the position and form of hands and fingers, where N contains various information, such as duration of key pressing, note strength and tone as well as pitch and note length.

Conversely, automatic fingering decision is considered as the problem of estimating the fingering S which an ideal performer would adopt most likely for the given sequence N of performed notes. Here, for the purpose of the model approximation described later, we factorize the sequence N of performed notes into a triplet sequence (Y, O, T ), where T is the sequence of note lengths, Y the sequence of pitches, and O the sequence of all other information which could include note strength and tone. Then the problem to be solved is to calculate the fingering S which maximizes P(S|(Y,O, T)) for a given sequence N = (Y, O, T ).

The most important is the probability whether the player can make a smooth transition from the current note to the next note under the state transition from the current state to the next state of hands and fingers. Assuming that the state of hands and fingers at each note ni = (yi, oi, ti) depends only on the state at its previous note ni-1 and its note length ti-1, the probability of fingering S being used for the sequence N = (Y,O, T) of performed notes is approximated by Bayes' Theorem as follows:

where P (·) denotes probability.

Consequently, an arc-emission HMM (Mealy machine) fit in very well as a fingering decision model where the states of the hands and the fingers are hidden states and a sequence of performed notes results from the state transition process.

The output probability function, when transiting from a state s1 to a state s2, is represented by bs1,s2 : M → [0, 1] where M is the set of all the possible note transitions. The function bs1,s2 indicates the likelihood of a given fingering transition s1 → s2 when playing the respective note transitions. For example, let s1 be a state using the 4th finger of the right hand and s2 the 5th finger, and consider the fingering transition from the 4th to the 5th. Then the fact that performance of F#→G is natural while that of A→G is not is shown in Figure 1.

2.3 Model Formulation through HMMs

By using HMMs, fingering decision can be looked as the probabilistic inverse problem of estimating the fingering hidden behind a sequence of performed notes, which is parallel to the principle of modern speech recognition. Fingering decision is thus formulated as the problem of calculating S which maximizes the a posteriori probability P (S|(Y, O, T )) in formula (1) with Viterbi search (see [Huang et al., 2001]).

As already stated, the main purpose of our paper is to probabilistically formulate the fingering-to-performance process which fits well with Hidden Markov Models (HMMs). Though often misunderstood, HMM represents a class of probabilistic structures where the learning process is not always essential. We consider physical constraints in fingering and give reasonable parameter values to examine our model.

Using this formulation, it is also possible to handle with expressive performances. Given an (imaginary) expressive music performance, the most likely fingering that has produced such a performance can be estimated according to the a posteriori probability maximization principle.

2.4 Model Approximation

The fingering decision model described above can typically contain a very large number of model parameters. In this paper, as a first step, we introduce the following three approximations to narrow down the parameters to a small number.

Simplifying the States of Hands and the Note Information

We suppose that a finger (or a set of fingers) playing a note (or a set of notes) at some point in time is the hidden state and that pitch transition (e.g., F#→G#) is the output. In other words, we ignore O and T . That is:

where fi is the finger (set of fingers) playing the ith note (set of notes). Through this approximation, the set M introduced in Section 2.2 is considered as the set of all the possible pitch transitions.

This approximation is intended to recognize the fundamental functionality of the decision model by limiting input/output data to the pitches (the primary input for fingering decision) and the fingers (the primary output to be decided). One can however expect the following drawbacks:

• Limiting the note information which affects fingering to the pitches might miss important factors of fingering decision (for example, the decision cannot take note lengths into account), and decision accuracy might thus decrease.
• It is impossible to decide the fingering in a broad sense (such as the position/form of the hands, elbow motion, and so on). Limiting the decided information to fingers position might narrow the range of possible applications of the algorithm (e.g., estimating the required playing skill of piano scores based on fingering information).

The former problem is expected to be solved by including note information as parameters of M in the function bs1,s2 : M → [0,∞). The latter is expected to be solved by defining possible fingerings (in a broad sense) at each point in time as a discrete set and defining state transitions inside this set.

Ignoring Octave Position

We ignore the octave position of pitches when defining the set M. This means that, for example, the pitch transition D→G performed in the middle of the keyboard is supposed to be the same as when performed in the right side of the keyboard, although the keys are not equally easy to play.

This approximation is intended to reduce the domain M of the output probability function bs1,s2 : M → [0,∞). However, the difficulty of playing low/high pitch keys in the right/left hand cannot then be considered in the fingering decision model, which will have a negative effect when deciding which hand has to be used to play a note. A possible way to deal with this problem is to adjust the output probability functions bs1,s2 for each octave position of the pitch transitions. This adjustment should be done for each bs1,s2 because the range of keys which can be easily played will differ according to the finger used.

Simplifying the Output Probability Function

We express a pitch transition in the set M as a twodimensional vector -→d from a key position to another in order to consider two-dimensional (right/left and back/fore) finger position on the keyboard. We suppose that the output probability function bs1,s2 : M → [0,∞) defined on the continuous space M follows a two-dimensional Gaussian distribution.

Figure 2 shows an example of such a distribution bs1,s2 . It shows that if F# (the origin of the plane) is played with the 3rd finger (= s1) of the right hand, then the note played next with the 5th finger (= s2) of the right hand is most likely to be A (the mean of the distribution).

This approximation is intended to decrease the number of parameters which enter into the definition of the function bs1,s2 : M → [0,∞) for a given hidden state transition s1 → s2.

However, this strongly constrains the form of the function bs1,s2 (for example, bs1,s2 cannot be locally adjusted), therefore accuracy improvement by parameter learning might be limited. A possible way to solve this problem is to relax the constraint in some way, for example by replacing the Gaussian distribution with a distribution in polygonal form.

2.5 SettingModel Parameters

It is necessary that not only the structure and the parameters of the fingering decision model but also the specific values taken by the model parameters reasonably explain the motion and the structure of hands and fingers in piano performance. Therefore, it is important to choose the values of the model parameters such that they reflect the physical structure of the human hand and the conventions in the actual fingering decision. For example, concerning the physical structure, the finger lengths and the tendency that the stretch between the 1st finger (the thumb) and one of the other fingers is easier than that among two of the 2nd, 3rd, 4th and 5th fingers should be taken into account, while for the decision conventions one should model the fact that the distance in a finger pair usually corresponds to the distance in a key pair but that finger-crossing sometimes occurs.

Specifically, we should decide the values of the model parameters based on the following ideas:

• The horizontal-direction element of the mean of the output probability function should be close to the distance in the key pair corresponding to the distance in the finger pair. For example, the transition from the 3rd finger to the 5th finger corresponds to twice the width of a white key.
• The depth-direction element of the mean of the output probability function should be close to the difference between the distances from the wrist to the fingertips.
• The tendency of the stretch between the 1st finger and one of the other fingers to be easier should be reflected by choosing a larger value for the variance of the corresponding output probability function.
• The fact that finger-crossing likely occurs in the transition between the 1st finger and one of the other fingers should be reflected by choosing a corresponding output probability function with two distinct peaks. The two peaks will correspond to normal fingering and fingercrossing respectively.

• The fact that the fingering making a transition between the 3rd(4th) and 4th(5th) fingers is often avoided because of the limited independency of those finger pairs should be reflected by choosing a smaller value for the corresponding state transition probability.

3 Experiments

This chapter describes the experiment performed to show the fundamental functionality of the proposed algorithm.

3.1 Conditions

We used piano scores of monophonic melodies in a single hand (specifically, the right-hand monophonic melodies in Bach's Inventions No. 1 and 2, which include about 620 notes in total). We made the three approximations introduced in Section 2.4. Here, larger amount of piano scores should be used in experimental evaluations and we plan to increase our sample data. Our current paper, however, focuses on formulating the fingering decision problem, leaving the parameter learning issues to our future study.

We initially set the values of the model parameters as described in Section 2.5, and then tuned them manually so that the decided fingering becomes as likely as possible. As the manual tuning of model parameters can be regarded as a primitive learning process, we suppose there is little risk of over-fitting where only a few parameters are involved in the manual adjustment.

3.2 Results

The decided fingering is reasonable in many places as can be seen in Figure 3:

• (A) The distance in the key pair corresponds to that of the finger pair,
• (B) Finger-crossing is introduced in appropriate places,
• (C) The difficulty of playing a black key with the 1st finger is reflected.

However, the decided fingering is problematic in a few places as shown in Figure 4:

• (D) It is difficult to play the same key consecutively with the same finger for some note lengths,
• (E) The distance in the key pair corresponds to that in the finger pair but the hand motion is awkward.

3.3 Analysis

The reasonable parts (A)-(C) in the decided fingering are considered to be resulted from an appropriate choice of model parameters described in Section 2.5. Meanwhile, the problematic places (D) and (E) are considered to be due to the approximations introduced in Section 2.4. Specifically :

• (D) The problem comes from the fact that note length information T is ignored in the sequence N = (Y, O, T ).
• (E) The problem comes from the fact that the set of hidden states is lacking of model parameters representing the hand motion.

The problem (E) may be also caused by the assumption made in Section 2.2 that the current state of hands and fingers does not depend on the states at the notes before the previous note.

We can see that the manually-adjusted model parameters are not far from the values given by the parameter setting guidelines described in Section 2.5 (the values are reproduced in Tables 1, 2 and 3). In particular, the tendency to often avoid some fingerings because of the limited independency of the involved finger pair is achieved through corresponding state transition probabilities which are lower by approximately 30 percents than other state transition probabilities.

4 Considerations on the Model

4.1 Characteristics of the Model

As mentioned in Section 2.2, our problem is to calculate the fingering sequence S which maximizes P(S|N) for a given sequence N of performed notes. In this expression we can easily see that the current fingering reflects the notes at all points in time. This can be seen from another viewpoint as follows.

In Section 2.2, we focused on the transition from the fingering state for one note to that for the next note, and made a minimal assumption in formula (1) that the fingering state at every note only depends on the state at its previous note and its note length. We select, by means of Viterbi search, the chain of such transitions throughout the entire score with a high a posteriori probability, hence we can see that the current fingering reflects the notes at all points in time.

Consequently, we can say that the model described in this paper formulates the mechanism of fingering decision from an essential and minimal hypothesis, and that maximum effect is obtained through the tuning of the HMM function and the use of very efficient existing methods such as Viterbi search and parameter learning.

4.2 Training the Model Parameters

Experiments still have to be performed on what values the model parameters will converge to through parameter learning. However, from the discussion in Section 3.3 on the result of the manually-adjusted model parameters, one could expect that the convergence values are near to the ones reflecting the physical structure of the human hand and the conventions in the actual fingering decision described in Section 2.5. Moreover, besides parameter learning based on score data, it might also be possible to perform parameter learning by measuring the physical structure of a performer's hands.

4.3 Fingering Decision Based onMusicality

The fingering decision algorithm proposed reflects the physical structure of the human hand and the conventions in the actual fingering, and the fingering decided is intuitive from the viewpoint of the physical motion of the human hand. However, earlier musicological discussions show that counterintuitive fingerings may be chosen to reflect phrasing or motif in music (for example, [Bamberger, 1976]).

We should first work on the fingering decision based on the physical motion of the human hand, which will clarify the limits of such decision and help us work further on the fingering decision based on musicality.

4.4 Estimating the Required Playing Skill

It is possible to calculate the likelihood value per note of the fingering decided by the model, which can be thought of as an expression of the required playing skill of a given piano piece. The value calculated in this way is characteristics of piano scores and expected to be a good feature of a piano piece. This may be used as retrieval criteria for piano piece retrieval or as means for classifying piano pieces according to their required playing skill.

5 Discussion on the Algorithm

The advantages of the proposed algorithm include:

• The mechanism of fingering decision can be expressed qualitatively and quantitatively by the characteristics and the values of the model parameters, for example, the stretchability of a finger pair with which two notes are played and the difficulty of moving a finger pair independently (see Section 2.5).
• Reasonability and likelihood of the estimation process and result can be expressed as probabilities. Conflicts among applicable rules do not arise as in rule-based approach.
• The algorithm can decide not only the fingers used but the position and the form of the hands and the fingers, based on which applied researches are expected such as estimation of the required playing skill.

Moreover, the proposed algorithm can be extended in the following ways:

• The algorithm can be applied not only to monophonic melodies in a single hand but also to chords and polyphonic melodies in both hands even in the case where the hand to be used is undecided. See the sample scores in Figure 5.
• Exceptional rules can be applied by limiting the search space, such as partial fingering instructions in scores or the rule that the 1st finger should be used for sforzando on a strong beat.
• Parameter learning is possible even from scores with no fingering information by means of Baum-Welch algorithm, as well as from scores tagged with fingering information. Moreover, parameter learning might be possible by measuring the physical structure of a performer's hands, such as their size or the stretchability and independency of finger pairs.
• Multiple-solution search can be realized by N-best search.

The proposed algorithm is expected to be applied to various technologies. We give several examples below.

• Application based on required playing skill: Estimating the required playing skill and generating practice planning from given scores. Piano piece retrieval using the degree of required playing skil as a retrieval key (for the estimation of the required playing skill, see 4.4).
• Application to composition and transcription: Judging whether an automatically-composed piano piece is actually playable. Basic technology for automatic composition and transcription of piano pieces which are both easy to perform and effective, such as transcribing from orchestral to piano pieces.
• Application to other instruments: Applying the mathematical model proposed in this paper to other instruments such as guitar.

6 Future Works

We are currently working on the implementation of a more accurate version of our model and of the parameter learning.

We also plan to extend our model to chords and polyphonic melodies in both hands including the case where the hand to be used is undecided. Considering 1024(= 210 ) states to represent all the combinations of the 10 fingers each of which is pressing a key or not, polyphonic fingering problem can be solved in a way similar to the monophonic case. In addition, we will work on comparative performance evaluations of other approaches and models, considering expressive piano performances.

7 Summary

This paper proposed an algorithm for automatic decision of piano fingering based on HMMs. Piano performance is modeled as a process by which a sequence of respective (hidden) state transitions produces a sequence of respective note transitions and the maximum a prosteriori solution is obtained by the Viterbi algorithm. The algorithm proposed has multiple advantages and can be extended and applied in many ways. The experiment based on the approximated model has shown the fundamental functionality of this algorithm.