Multiple Instance Learning-Based Birdsong Classification Using Unsupervised Recording Segmentation

1 Introduction

One of the biggest challenges in ecology and conservation biology is the assessment of biodiversity through effective monitoring techniques that allow covering large scales in both time and space [Depraetere et al., 2012]. Nevertheless, traditional methods for assessing the presence and abundance of species are expensive as well as spatially and temporally limited since they typically consist in manual data collection through extensive transects [Aide et al., 2013]. Such a task would be much easier by using technological tools, e.g., it is possible to implement a continuous monitoring of the soundemitting wildlife -as birds, which have been traditionally used as a biodiversity indicator [Briggs et al., 2012]- by installing microphones and recorders in the field. Moreover, in regions with high cloudiness, this technique may be even more effective than visual inspection.

Signal analyses and classification approaches used in bioacoustics range from trained humans listening to recordings or visually inspecting spectrograms, to autonomous classification systems based on digital signal processing and pattern recognition methods [Blumstein et al., 2011]. However, considering the overwhelming amount of acoustic data that can be collected, relying on analyses made by human experts is limited and often unfeasible. Furthermore, according to [Hao et al., 2012], the automation of analyses using signal processing techniques and pattern recognition algorithms is less expensive -in the long term- than the assessments made by experts and, potentially, even more accurate.

As indicated by Briggs et al. [2012], methods for acoustic classification of bird species can be categorized into two types: 1) those that classify individual syllables, and 2) those that classify recordings having sounds of multiple sources. The former ones require a detailed annotation of each segment while in the latter ones, for training, it is only required to label the presence of the species of interest in each recording. In either case, segmentation is a high-priority step [Neal et al., 2011]. However, methods of the first type are more sensitive to the segmentation quality because omitted syllables become false negatives and those sounds incorrectly detected could become false positives. In contrast, for methods of the second type, it is possible to achieve correct classification even if the two above-mentioned miss-segmentation cases occur since every recording may contain multiple syllables.

In general, segmenting recordings into smaller recognition units is assumed as a part of the preprocessing stage and it is done either manually or automatically. Yet, automatic recognition should not require manual segmentation [Trifa et al., 2008]. To this end, segmentation algorithms have been developed mostly using energy and entropy as criteria to identify onset and offset times of the regions of interest [Fagerlund, 2007]. Under ideal conditions, when the vocalization call is the only sound in the recording, an increase in energy clearly reveals a region of interest, making segmentation procedures simple enough [Neal et al., 2011]. In real conditions, however, recorded signals are degraded due to the presence of many sound sources, e.g., wind streams, background noise from other animals and surrounding events. In spite of that, several research studies on automated species recognition clarify that their methods work well when the recognition units are correctly detected, often this issue is not discussed in depth (making only a brief description). Furthermore, as indicated in [Hao et al., 2012], it should be taken into account that achieving perfectly segmented data is at least as difficult as the classification step.

Considering that the recognition of recordings has the advantage of not making the impractical assumption of requiring perfectly segmented data, in this study, it is proposed a classification methodology for audio recordings which only uses -in the training phase- labels from training recordings, that is, isolated and labeled vocalization segments are not required beforehand since the methodology includes a novel unsupervised segmentation method for birdsong recordings. Interest sounds are detected from the Shorttime Fourier transform (STFT). In the segmentation method described in Sec. 2.1, the output is a matrix of the same size of the corresponding spectrogram, where interest elements (or pixels) are marked with "one" and non-interest elements with "zero". Classification is carried out using the multiple instance classification (MIC) approach, as follows: 1) neighboring interest pixels are grouped into regions, 2) each region is described by a feature representation and 3) a classifier based on multiple instance learning (MIL) is trained considering each spectrogram as a bag of instances. Its classification performance is estimated when using the unsupervised segmentation method proposed in Sec. 2.1 and compared against the performance obtained when using the supervised segmentation method proposed by Briggs et al. [2012]. Both methods consist in the detection of regions in the spectrogram likely associated with vocalizations; however, in the latter, it is required to provide a set of manually annotated spectrograms where pixels have been labeled according to whether or not they correspond to bird sounds.

2 Material and methods

2.1 Segmentation of birdsong recordings

Time-frequency analysis of audio recordings is usually carried out through spectrograms representing power intensity at each time-frequency point. Particularly, spectrograms are considered as recognizable images to identify bird species [Dennis et al., 2011], whose vocalizations are represented by intensity variations. Thus, under the assumption that segment vocalizations give a form of continuous regions holding the highest power values, our unsupervised segmentation methods consists in the following stages (see Fig. 1):

• Spectrogram estimation: Based on the STFT decomposition, we compute a spectrogram matrix S={sij :

.......................................................................................................................................................................................

• i = 0, . . . , N - 1; j=0, . . . ,M - 1}, with S∈RN×M , where indexes i and j stand for the frequency and time domains (i.e., N points in the frequency domain that are estimated in each one of M time frames). We use the Hann window lasting 512 samples and overlapping 256 samples as in [Briggs et al., 2012].
• Preprocessing: After applying the two-dimensional Wiener filter, a denoised and smoothed spectrogram, ˜ S={˜ sij} is estimated with elements:

where µ ∈ R and σ2 ∈ R are the local mean and variance, respectively, and σ2 η ∈ R is the noise variance. The first two values are estimated at a k×k neighborhood centered on each point (we set k=5 as in [Pourhomayoun et al., 2013]), and σ2 η is the average over all local variances σ2 . Then, we suppress structures that are lighter than their surroundings and are connected to image borders by considering a value of 8 as connectivity parameter. An erosion [van den Boomgaard and van Balen, 1992] is performed before by estimating the lowest number of image elements that are not connected to the edges. The erosion procedure results in a smoothed image I ∈RN×M , from which we perform morphological reconstruction of ˜ S to remove all intensity fluctuations (except the intensity peak). As a result, we get the matrix H ∈ RN×M that only holds objects with neighboring borders. At the same time, the difference matrix, ˜ H = ˜ S-H is also computed holding only those objects from the original image not having neighboring borders.

• Thresholding: We fix a threshold to binarize each image ˜ H using the nonparametric and unsupervised Otsu's method of automatic threshold selection [Otsu, 1979]. Extracted only from a computed gray-level histogram, the optimal threshold, To, is selected by maximizing an introduced discriminant measure of separability among all resultant gray level classes, as follows:

where φT = ∑Lj=1 jpj, φ(k) = ∑kj=1 jpj, ω(k) = ∑kj=1 pj, pj =nj/N are the values of the normalized gray level histogram, L is the number of gray levels, nj is the number of pixels at level j, and N is the total number of pixels over the whole difference image ˜ H. Therefore, the optimal threshold is computed as To = (ˆ k - 1)/(L - 1).

• Mask operation: To select the most relevant pixels from the spectrogram at hand, a binary matrix B = {bij}, with B∈RN×M , is obtained by thresholding as follows:

• Instance vector extraction: From computed arrangements B and S, we compute a spectrogram region set R = {Ri : i = 1, . . . , l}, holding the respective matrices of all-connecting points Ri ∈ RNi×Mi , with Ri ⊂ S, being Ni < N and Mi < M. The number of regions l is automatically fixed as indicated in [Zakaria et al., 2012]. Lastly, each region Ri supplies a single feature vector (or instance), denoted as xi ∈ Rd . The number of features d is fixed in accordance with the training scenarios explained in Sec. 3.1.

2.2 Multiple instance classification

Within the standard supervised classification framework, the training set consists of n feature vector examples or instances X = {xi ∈ Rd : i = 1, . . . , n} and their labels, in a twoclass problem, Y = {yi ∈ {0, 1} : i = 1, . . . , n}. Thus, any classifier function, X → Y, is trained to predict labels for each novel instance.

In MIC, an object is represented by a set, or bag, Xi = {xij ∈ Rd : j = 1, ...,mi} of mi instances xij and a label ˜ yi , i.e., each label is associated with the entire bag but labels of the individual instances are unknown. Therefore, the training set consists of n bags ˜ X = {Xi ∈ Rmi×d : i = 1, ..., n} and their corresponding labels ˜ Y = {˜ yi ∈ {0, 1} : i = 1, . . . , n}. Then, the classifier function, ˜ X → ˜ Y, is trained to predict labels for each novel bag of instances. We address the problem of classifying recordings using the MIC approach because, in our case, we have labels for recordings, i.e. bags, but they are represented for several instances (feature vectors extracted from each region detected after the segmentation stage).

Particularly, we use the MILES (MIL via Embedded Instance Selection) classification algorithm because it has been experimentally shown that it performs well with bioacustic signals [Cheplygina et al., 2015]. MILES transforms the original MIC problem into a standard supervised learning framework injectively relating instances and labels [Chen et al., 2006]. It maps each bag into a feature space defined by instances in the training bags using an introduced instance (dis)similarity measure. Thus, bags are represented by the maximum (dis)similarity to all other instances. On this (dis)similarity representation, a sparse linear classifier is trained [Tax, 2013].

2.3 Segmentation performance measures

Since the manual recording segmentation is a very fatiguing task, rather than directly comparing between automated and manual outputs we indirectly estimate the quality of the proposed segmentation method by the recording classification performance that must be strongly influenced by the used segmentation procedure, as discussed in [Briggs et al., 2012]. The most common performance measures for a classifier are the following ones: accuracy a = (TP +TN)/(P+N), specificity s = TN/N, recall rate r = TP/P and precision rate p = TP/(TP + FP ); where TP is the number of recordings correctly classified as positives, TN is the number of recordings correctly classified as negatives, FP is the number of recordings incorrectly classified as positives, FN is the number of recordings incorrectly classified as negatives, P and N are the total number of positives or negatives recordings, respectively.

However, these performance measures are affected by the relative size of the classes. Therefore, to overcome that drawback, the two-class F-score is used as the performance measure defined as follows:

The F-score ranges from 0 to 1 where the higher its value, the better the classification performance. In this work, the one-against-all reduction frommulti-class task to binary classification technique is used where each species is selected as objective (positive) class since the F-score is a two-class measure. As regards the classifier performance, we carry out validation using the Bootstrapping technique where input audio data are randomly split into two sets: one-half for training and one-half for testing. This procedure is carried out ten times.

As suggested in [Bramer, 2013], each one of the eight considered feature sets over the 13 data sets (see Sec. 3.2), are compared in terms of the paired t-test, for which the null hypothesis states that the performance of two classification strategies can be statistically assumed as the same. The pseudo-code to determine whether to accept the null hypothesis is presented in Algorithm 1. Otherwise, when the null hypothesis is rejected, we select as the best strategy the one having the highest pair performance difference computed in average over all considered classification strategies.

3 Experimental Set-Up

3.1 Training strategy

Figure 2 shows the training scheme used through all experiments to test the proposed birdsong classification methodology based on unsupervised segmentation of audio recordings and multiple instance learning.

In the segmentation stage, only one approach, either the unsupervised (see Sec. 2.1) or the supervised (proposed in [Briggs et al., 2012]), is used. Besides, the following four representation scenarios are separately considered: 1)

Algorithm 1 Process to determine whether or not to accept the null hypothesis of the paired t-test. Based on [Bramer, 2013].

• 1: Let z = [z1, z2, ... zn] be the vector of the differences in classification performances between strategies 1 and 2 and let n be the number of data sets.
• 2: Assign tlevel considering the following: for 13 degrees of freedom, t ≥ 1.771 would only be expected to occur by chance with probability 0.10 or less, and it is said that the hypothesis is rejected at 10% level. Therefore, if tlevel = 1.771, tlevel = 2.160 or tlevel = 3.012, and |t| > tlevel, the null hypothesis is rejected at the 10%, 5% or 1% level, respectively.
• 3: procedure T-TEST(z, n, tlevel)
• 4: Compute a = (∑i=n i=1 z2 i ) - (∑i=n i=1 zi)2 /n.
• 5: The sample variance s2 is equal to a/(n-1).
• 6: The sample standard deviation is the square root of s2 .
• 7: Divide s by the square root of n to get the standard error estd.
• 8: The t statistics is computed dividing the average value of z by estd.
• 9: if |t| ≥ tlevel then
• 10: Return The hypothesis is rejected.
• 11: else
• 12: Return The hypothesis is assumed as true.
• 13: end if
• 14: end procedure

Mask descriptors, denoted as "MD", that describe region shape; the following set of features are computed: minimum frequency, maximum frequency, bandwidth, duration, area, perimeter, non-compactness, and rectangularity. 2) Profile statistics, "PS", a set of fourteen features are computed, which are based on statistical segment properties in time and frequency: frequency-Gini, time-Gini, frequencymean, frequency-variance, frequency-skewness, frequencykurtosis, time-mean, time-variance, time-skewness, timekurtosis, frequency-max, time-max, mask-mean, and maskstandard deviation. 3) Histogram of Gradients, "HOG", this set consists of 16 features characterizing shape and texture of each region where gradient directions over the pixels of the region are computed; each histogram holds 16 bins equally spaced over the angle range [-π, π] and features are extracted from the normalized 1-D histograms [Dalal and Triggs, 2005]. 4) All-features set, "AF", that merges all above feature sets into a single one, as in [Briggs et al., 2012].

Therefore, eight training strategies are tested for the feature extraction stage. In case of supervised segmentation training, the affix "-Br" (meaning Briggs) is added to the end of every feature notation, in accordance to the segmentation method proposed by Briggs et al. [2012]. Lastly, we use the MILES classification algorithm. We employ an exponential kernel exp{-(||a - b||)/p} as the instance (dis)similarity function, where parameter p ∈ [2 . . . 5] is heuristically fixed and notation ‖ · ‖ stands for Euclidean-norm.

3.2 Data set

For the sake of comparison, we perform experiments with the publicly available 1 data set used in [Briggs et al., 2012]. This data collection holds 548 recordings sampled at 16 kHz that were manually labeled. The data set contains 13 bird species, often vocalizing simultaneously and perturbed with environmental noise, though each recording lasting ten-seconds holds between one and five species. Table 1 shows the amount of recordings holding each considered species.

3.3 Results of compared classification strategies

Table 2 shows the estimated F -scores for the different considered feature sets; notice that the HAFL data set has the highest F-score (> 0.99). In contrast, both data sets DEJU and HETH achieve the lowest values (< 0.21) and get zero-value F-score for several features because those data sets hold very few recordings (see Table 1): 20 and 15, respectively.

In terms of the considered feature sets, the use of AF-Br reaches the highest average performance value (F = 0.821) that is averaged over all considered objective classes, while HOG gets the lowest one, F = 0.723, for the baseline supervised approach. Once again, AF (F = 0.812) and HOG (F = 0.706) are the best and worst cases, respectively, for the unsupervised method. However, the MD feature set, in average, benefits the most from the use of the unsupervised segmentation, while the HOG set degrades the worst. It must be noted that the average performance is taken without DEJU and HETH data sets because of their accomplished anomaly values. This situation may be explained since the MD feature set encodes region shape attributes that are far from being easy to be manually computed, as seen in Fig. 3(a) showing a typical birdsong spectrogram. In contrast, the texture-based HOG set implies calculation of gradient directions over region pixels (see Fig. 3(b)); this procedure includes enhanced Gaussian filtering and histogram binning that are very sensitive to their parameter tuning.

1 Audio recordings of the data set are available at http://www.miproblems.org/datasets/birds/

In order to provide an intuitive illustration, all F -score values estimated in Table 2 are graphically presented in Fig. 4 where the vertical axis represents the performance obtained by the proposed unsupervised-based segmentation method, while the horizontal axis corresponds to the one obtained by the supervised reference method. Since the larger the F -score - the better the performance, any point above the diagonal line indicates that the proposed method outperforms the reference. In most of the cases, no remarkable difference in terms of performance is observed between both segmentation methods tested for the same representation scenario.

In case of the worst performance when the classifier guesses at random with equal frequency (i.e., Tp = P/2, Fn = P/2, FN = N/2, and Tn = N/2), then one may calculate a threshold Ft = P/(1.5P +0.5N) value, under which the classifier is just guessing. In our case, Ft = 0.42 that is achieved for the BRCR data set. As shown in Fig. 4, most of the computed F-scores overcome by far the Ft threshold.

Table 3 shows several performance measures (namely, recall, r, specificity, s, precision, p, accuracy, a, and the Fscore) obtained for the whole feature sets, i.e., AF and AF-Br data sets. Typically, both training strategies may be differently influenced by each considered class. Particularly, the former training strategy gets the better specificity for DEJU, HETH, VATH, HAFL, and WETA classes, while the latter strategy instead of WETA the HETH class also gets the highest value. At the same time, only the HALF class remains the best in terms of accuracy.

To provide a better illustration, Table 4 shows the obtained results of the paired F-score difference, ∆F, computed between the AF and AF-Br features. Since the latter set is the reference, the estimated ∆F value gets negative sign when the reference feature set is better, otherwise ∆F becomes a positive value. The best and worst achieved ∆F values are marked in bold. Particularly, the SWTH gets the lowest difference (-0.169), that is, that class is the most negatively influenced by the proposed strategy while the CBCH class achieves the best influence (0.084). However, though the average value is ∆F = -0.017 in support of AF-Br set, the corresponding estimated t value gets as high as 0.905, meaning that neither of the considered feature sets are statistically different at levels of 10%, 5%, and 1%.

Lastly, the null-hypothesis values are shown in Table 5 in order to make clear the influence of each considered segmentation strategy, in terms of performed F-score classification measure. Values are computed at 10% level (a strong assumption) to either admit (value 0) or deny (value ±1) the null hypothesis about their statistical similarity between each pair of contrasted feature sets (columns stand for the reference supervised feature sets and rows for the proposed unsupervised sets). In case the statistical similarity is rejected, the null hypothesis gets the value 1 if the column-wise feature set reaches a better performance than the the row-wise set. Otherwise, the null hypothesis gets the value -1.

As seen from the obtained main diagonal matrix values, one can infer that each compared feature set, except MD, has no statistical difference regardless of the used segmentation approach. In case of the MD set, its unsupervised version turns out to be better. This situation should be expected due to the above-given explanation about the advantage of automated MD feature extraction. As a result, the proposed birdsong classification methodology based on unsupervised segmentation of audio recordings and multiple instance learning has no statistical difference with the baseline supervised version, at least, when using the three extracted feature sets: PS, HOG, and AF.

4 Discussion and Concluding Remarks

In this paper, the use of unsupervised segmentation of audio birdsong recordings is investigated along with multiple instance learning to classify among a given number of bird species. The proposed unsupervised segmentation of audio birdsong recordings is contrasted against its baseline reference supervised version requiring manual annotation of properly computed spectrograms, as described in [Briggs et al., 2012]. Yet, since this manual recording segmentation poses as a very fatiguing task, we indirectly estimate the quality of the proposed segmentation method by the introduced two-class F -score as classification performance measure that is not affected by the relative class size. Afterwards, each one of the considered feature sets are compared in terms of the paired t-test, for which the null hypothesis states that the achieved F -score performance of two given classification sets can be statistically assumed as the same. The univariate paired t-test is preferred due to its simple interpretation though other multivariate tests may be used, for example, the multivariate paired Hotelling's T 2 test that provides similar results in our work. Both segmentation approaches are validated for the four feature sets: MD, PS, HOG, as well as the all-features set. In average, the MD feature set benefits the most from the use of the unsupervised segmentation, while the HOG set degrades the worst, as seen in Table 2. The main reason for the latter results is the fact that parameter tuning of the automated HOG feature extraction should be improved. However, this procedure is out of the scope of the present work. Nonetheless, according to the accomplished values of the null-hypothesis test shown in Table 5, the introduction of the unsupervised segmentation of audio recordings has no statistical difference with the baseline supervised version, at least, when using the following three extracted feature sets: PS, HOG, and AF.

This work provides a birdsong recognition framework using the MILES classifier, which in turn uses an exponential kernel as instance (dis)similarity measure. Even though performed F-score values are high, this classifier is sensitive to a low number of training recordings. As a conclusion, the proposed unsupervised recording segmentation of audio birdsong recordings improves species classification with the benefit of easier implementation since no manual handling of recordings is required, making feasible the design of fully automatic birdsong recognition systems.

As future work, the authors consider that more studies must be undertaken on improving the feature extraction stage. Besides, the use of classifiers demanding less input training sets remains an important issue since, in practice, collecting labeled audio birdsong recordings is very costly.

Acknowledgment

This work is supported by "Programa de formaci´on doctoral en Colombia" COLCIENCIAS and "Programa de Internacionalizaci ´on del Conocimiento - Modalidad 3" from Universidad Nacional de Colombia.