# LC-Protonets: Multi-label Few-shot learning for world music audio tagging
**Authors**: Charilaos Papaioannou1,21,2} 1,2, Emmanouil Benetos22} 2, Senior Member, Fellow, IEEE
> School of ECE, National Technical University of Athens, Greece Centre for Digital Music, Queen Mary University of London, UK
XX Month, XXXX XX Month, XXXX XX Month, XXXX XX Month, XXXX XX Month, XXXX XXXX.2022.1234567
Corresponding author: Charilaos Papaioannou (email: cpapaioan@mail.ntua.gr).
## Abstract
We introduce Label-Combination Prototypical Networks (LC-Protonets) to address the problem of multi-label few-shot classification, where a model must generalize to new classes based on only a few available examples. Extending Prototypical Networks, LC-Protonets generate one prototype per label combination, derived from the power set of labels present in the limited training items, rather than one prototype per label. Our method is applied to automatic audio tagging across diverse music datasets, covering various cultures and including both modern and traditional music, and is evaluated against existing approaches in the literature. The results demonstrate a significant performance improvement in almost all domains and training setups when using LC-Protonets for multi-label classification. In addition to training a few-shot learning model from scratch, we explore the use of a pre-trained model, obtained via supervised learning, to embed items in the feature space. Fine-tuning improves the generalization ability of all methods, yet LC-Protonets achieve high-level performance even without fine-tuning, in contrast to the comparative approaches. We finally analyze the scalability of the proposed method, providing detailed quantitative metrics from our experiments. The implementation and experimental setup are made publicly available, offering a benchmark for future research.
Few-shot learning, Prototypical networks, Multi-label classification, Audio tagging, World music datasets.
## 1 INTRODUCTION
Humans are capable of learning from just a few examples [1], highlighting a significant difference between human and typical deep learning at present. Few-shot learning aims to bridge this gap by introducing methods that can learn from limited data [2]. This approach is particularly significant for domains and tasks where large-scale data is not readily available, such as in the field of Music Information Retrieval (MIR) when studying under-represented music cultures.
Several systematic approaches have been undertaken in recent years to enrich MIR datasets with music from diverse cultural backgrounds [3]. In most cases, the produced datasets contain tags with various semantic information (e.g., instruments, genre) associated with each music piece. The task of predicting these tags is called automatic music tagging, or auto-tagging [4, 5, 6, 7], and constitutes a multi-label classification problem. These tags often exhibit a highly imbalanced distribution, with a small subset occurring frequently while many appear only rarely.
Few-shot learning [8, 9, 10] can enhance music understanding by addressing two key challenges. First, unlike certain transfer learning methods that are limited to frequent classes [11, 12, 13], it enables the inclusion of under-represented tags within well-established music domains, where annotated data is often sparse. Second, it supports the exploration of under-represented domains, such as new or emerging music cultures, where annotated datasets are even more limited.
In order to effectively address the automatic audio tagging problem, the adoption of multi-label few-shot learning (ML-FSL) methods is essential. While these methods have been employed in recent audio classification research [14], to the best of our knowledge, they have not yet been applied to MIR datasets. Such a technique can, in turn, be combined with either foundation models [15] or contrastive learning models [16, 17, 18], mapping their general-purpose representations to semantic spaces for more comprehensive music analysis.
The goal of this work is to develop an approach to enable multi-label classification for diverse music cultures, in the presence of imbalanced tags and limited data. The main contributions are as follows:
- We propose a multi-label few-shot learning method, “LC-Protonets”, that is based on creating multiple prototypes using the union of the power sets of the support samples’ labels.
- LC-Protonets are applied to the problem of automatic audio tagging for diverse music cultures, and are shown to outperform other multi-label few-shot learning approaches in the literature. Further, the results are consistent across datasets.
- We demonstrate the efficacy of a two-step learning methodology aimed at augmenting the inclusion of labels within imbalanced datasets.
- The implementation, along with the experimental setup, is provided in a public repository https://github.com/pxaris/LC-Protonets that can serve as a benchmark for similar research works.
## 2 RELATED WORK
### 2.1 AUTOMATIC WORLD MUSIC TAGGING
In order to describe a piece of music with metadata, various aspects can be considered, including genre, mood, and instrumentation. It is a common approach in the MIR field to regard this non-exhaustive list of metadata as tags that characterize a music work [19]. The task of predicting the tags of a music piece given its audio signal is referred to as auto-tagging [4, 5, 6, 7], constituting a multi-label classification problem, where an instance can be annotated with a set of multiple labels.
Several architectures have been proposed to address the automatic music tagging problem. Convolutional-based models, such as VGG-ish [20] and Musicnn [21], are commonly used to extract knowledge from time-frequency input representations, while architectures like SampleCNN [22] and TCNN [23] process the raw audio signal. The Audio Spectrogram Transformer (AST) was introduced and used for audio classification in [24]. In a recent work aimed at building an advanced foundation model capable of performing several tasks, including auto-tagging [25], a BERT-style encoder from HuBERT [26] and a Conformer [27] were employed.
Numerous research efforts in the MIR field have incorporated world music datasets [28]. In [29, 30], the authors conducted automatic makam recognition, while [31] used deep learning models to classify Indian art music. Research in [32] and [33] used pitch histograms to analyze Iranian dastgāhi and Korean folk music, respectively. A recent study [11] applied auto-tagging to diverse music datasets, but this approach was not designed to handle under-represented tags.
### 2.2 FEW-SHOT LEARNING
Few-shot learning (FSL) methods [8, 9, 10] aim to extract knowledge from limited example instances. These methods have been successfully applied to classification tasks across various domains, including computer vision [34, 35], natural language processing [36, 37], and acoustic signal processing [38, 39]. In the MIR field, FSL methods have been employed for tasks such as drum transcription [40], source separation [41], and single instrument recognition [42]. To the best of our knowledge, this is the first time FSL methods have been applied to multi-label classification within an MIR task, which represents a significant advancement in the field.
The core concept of conventional FSL involves classifying an instance, referred to as the query item, based on similarities calculated from class representations derived from a small set of examples called the support set. While this framework works well for multi-class tasks, where each instance belongs to a single class, extending it to multi-label classification presents additional challenges, as each instance can be associated with multiple labels.
Several methods have been proposed to address these challenges. In [43], a technique was introduced to synthesize samples with multiple labels by combining pairs of examples in the feature space. In [44], a module was used to predict the number of labels assigned to an item, which allows for multi-label predictions. Attention-based mechanisms were explored in [45], and a taxonomy hierarchy between tags was utilized in [14]. Another strategy, “One-vs.-Rest” [46], reformulates the multi-label problem into multiple binary classification tasks using different classifiers.
While these methods effectively address multi-label classification, they often introduce additional complexity during training, such as the generative module in [43], the label count module in [44], or complex episode formation in [46]. We argue that such complexities can be simplified or omitted, streamlining the training process and potentially improving performance.
## 3 PROPOSED METHOD
In this section, we first describe the Prototypical Networks method and its adaptation to the multi-label setting, followed by the introduction of our proposed method, “LC-Protonets”.
### 3.1 PROTOTYPICAL NETWORKS
Prototypical networks [8] are widely used in few-shot learning (FSL) and function by computing a prototype for each class, which represents the average embedding of the support items belonging to that class. Let $S$ denote the support set, consisting of $N× K$ examples, where $N$ is the number of unique classes (referred to as the N-way) and $K$ is the number of examples per class (referred to as the K-shot). The prototype for a class $c$ , denoted as $p_c$ , is computed as the mean of the embedded support examples for that class:
$$
p_c=\frac{1}{K}∑_(x_i,y_i)∈ Sf_θ(\mathbf
{x}_i)·\boldsymbol{1}_y_{i=c}, \tag{1}
$$
where $f_θ(x_i)$ represents the embedding of input $x_i$ through a mapping model, and $\boldsymbol{1}_y_{i=c}$ is an indicator function that equals 1 if the label $y_i$ of $x_i$ belongs to class $c$ .
Once the prototypes are computed, a query set $Q$ consisting of unseen examples is used to test the model. Each query item is classified based on the similarity to the prototypes, typically using Euclidean or cosine distance. Specifically, the query sample $x_q∈ Q$ is assigned to the class whose prototype is the closest in the embedding space:
$$
\hat{y}_q=\arg\min_cd(f_θ(x_q),p_c), \tag{2}
$$
where $d$ refers to the chosen distance function. During training, cross-entropy loss and a Softmax function over the computed distances are used to learn the embeddings.
Episodic learning: Prototypical networks are trained using episodic learning, progressing through N-way K-shot episodes. In each episode, $N$ classes are randomly sampled, and $K$ support examples are drawn for each class to form the support set $S$ . The query set $Q$ consists of additional examples drawn from the same $N$ classes. The model computes the prototypes from the support set, and the loss is calculated based on the classification accuracy of the query set. This episodic approach encourages the model to generalize better in few-shot settings by simulating small training tasks during learning.
Extension to multi-label setting: We adopt the term “ML-PNs” (multi-label Prototypical Networks) to refer to the extension of Prototypical Networks for the multi-label setting. This method follows the extension published in [14], where it is referred to as “Baseline”; however, we prefer a more explicit name here to enhance clarity. In this setting, where each sample may belong to multiple classes, each support item contributes to multiple prototypes. Let $y_i$ be the set of labels for a given sample $x_i$ . For each label $y_i,j∈y_i$ , where $j$ ranges from $1$ to the number of classes $N$ , the embedding $f_θ(x_i)$ is used to update the prototype corresponding to $y_i,j$ . Consequently, the prototype for each class $c$ is computed by averaging the embeddings of all support examples that belong to class $c$ , even if they have additional labels.
In this setting, the Softmax function is replaced by a Sigmoid function, allowing the model to predict multiple labels for each query item. Binary cross-entropy loss is then used to optimize the model:
$$
L=-∑_q∈ Q∑_cy_q,c\log(\hat{y}_q,c)+(1-y_q,c)\log(1
-\hat{y}_q,c), \tag{3}
$$
where $y_q,c$ represents the true label for class $c$ for query $q$ , and $\hat{y}_q,c$ is the predicted probability for that class.
### 3.2 LC-PROTONETS
Adapting few-shot learning to the multi-label regime presents a significant challenge, particularly because classes are correlated and each sample may belong to multiple classes. To address this, we propose Label-Combination Prototypical Networks (LC-Protonets), an approach to multi-label classification that extends Prototypical Networks in a simple yet effective way.
<details>
<summary>extracted/5860914/figs/LC_Protonets.png Details</summary>
### Visual Description
## Concentric Circle Diagram: LC-Protonets Structure
### Overview
The diagram illustrates a hierarchical structure of labeled sets (A-F) organized in concentric circles around a central point labeled "q". Each circle contains colored nodes representing combinations of letters, with a legend at the bottom mapping colors to specific sets. An arrow from "q" points to the {C, E} node, suggesting a directional relationship.
### Components/Axes
1. **Central Point**:
- Labeled "q" (black dot)
- Connected via arrow to {C, E} node (yellow)
2. **Concentric Circles**:
- 5 layers of dotted circles
- Each layer contains colored nodes labeled with letter sets
3. **Legend**:
- Located at bottom
- Color-coded sets:
- Blue: {A}, {A, B}, {A, C}, {A, B, C}
- Orange: {B}, {B, D}, {B, D, F}, {B, F}
- Yellow: {C}, {C, E}
- Purple: {D}, {D, F}
- Mixed colors: {B, D}, {D, F}, {C, E}
### Detailed Analysis
1. **Node Placement**:
- Single-letter sets ({A}, {B}, {C}, {D}, {E}, {F}) appear in outer layers
- Multi-letter sets ({A, B}, {B, D}, etc.) occupy inner layers
- {C, E} (yellow) is centrally positioned, receiving direct attention from "q"
2. **Color Consistency**:
- Blue nodes consistently represent sets containing "A"
- Orange nodes represent sets containing "B"
- Yellow nodes represent sets containing "C"
- Purple nodes represent sets containing "D"
- Mixed-color nodes (e.g., half-blue/orange) represent combined sets
3. **Spatial Relationships**:
- Nodes with more letters appear closer to the center
- {A, B, C} (blue) is the most inclusive set in the diagram
- {B, D, F} (orange) and {C, E} (yellow) are positioned at mid-layers
### Key Observations
1. **Hierarchical Organization**:
- Single elements form the outermost layer
- Combinations form progressively inner layers
- Central {C, E} suggests a pivotal relationship
2. **Color-Coding Logic**:
- Primary colors represent base elements
- Secondary colors (mixed) represent combinations
- No overlap in color assignments between sets
3. **Directional Flow**:
- Arrow from "q" to {C, E} implies causation or dependency
- No other directional indicators present
### Interpretation
This diagram appears to represent a conceptual framework for LC-Protonets, showing:
1. **Elemental Relationships**: The concentric circles demonstrate how individual elements (A-F) combine into complex sets
2. **Central Dependency**: The {C, E} node's central position and connection from "q" suggests these elements form a critical subsystem
3. **Support Structure**: The bottom layer's "Support items" likely represent foundational components that enable the higher-level combinations
4. **Exclusion Principle**: Each set maintains distinct color coding, implying mutually exclusive relationships between different combinations
The structure suggests a system where basic elements (A-F) combine in specific ways to form functional units, with {C, E} playing a central role in the overall architecture. The color-coding provides an immediate visual reference for set composition, while the concentric layout reveals the complexity progression from simple to compound elements.
</details>
Figure 1: The labels of the support set consisting of four items (bottom) and the LC-classes created from them (top). The concentric circles with the dashed lines pass through LC-Protonets with same representations in the embedding space that are graphed equidistant from the query item $q$ .
We consider multi-label classification as a problem where every combination of labels is a descriptive label. These combinations are all subsets of the label sets found in the support data, including the full label sets themselves. Hence, a support item $(x_i,y_i)$ with $y_i=\{A,B,C\}$ , defines and contributes to all the label combinations derived from the power set $P$ of $y_i$ , excluding the empty set:
$$
\begin{split}P(\{A,B,C\})=\{\{A\}&,\{B\},\{C\},\{A,B\},\\
&\{A,C\},\{B,C\},\{A,B,C\}\}\end{split} \tag{4}
$$
Figure 1 illustrates an indicative example. The support set $S$ consists of four items, each associated with a distinct set of labels. The set of label combinations, henceforth referred to as LC-classes, is defined as the union of the power sets of each $y_i$ in the support set. Each LC-class is represented by an LC-Protonet (LCP), whose representation is computed by averaging the embeddings of the support items that include the corresponding LC-class in their power set, as shown in the color-coded example in the figure.
This approach addresses the multi-label classification problem as a mixture of few-shot and zero-shot learning scenarios. For instance, in Figure 1, the $\{B,D\}$ LC-class does not directly correspond to any support item, but its representation is inferred from items (one in this example) that include it in their label power sets. This introduces a zero-shot learning aspect. Meanwhile, the $\{B,D,F\}$ class has one corresponding item in the support set, enabling a few-shot scenario.
For a query item $q∈ Q$ , the distances to all LC-Protonets are computed. Figure 1 provides a conceptual 2D representation of the embedding space, where concentric circles indicate equal distances from $q$ to different LCPs. In the actual space, the LC-Protonets with identical representations will occupy the same point.
In cases where a query item has equal distances to multiple LC-Protonets, the one representing the largest number of labels is selected. That way, hierarchical relationships and strong correlation between the labels are supported. In the depicted example, both the $\{E\}$ and $\{C,E\}$ LCPs have the minimum distance to $q$ , and the query is assigned to the $\{C,E\}$ LC-class. If $E$ is hierarchically subordinate to $C$ , the LCPs for $\{E\}$ and $\{C,E\}$ will share the same representation, but the model will consistently select the $\{C,E\}$ class. Even if $C$ and $E$ are not part of a formal hierarchy but they co-occur in the same support items, it is rational to assume strong correlation between them and, again, assign the query to the $\{C,E\}$ class.
Training phase: Training is conducted using an episodic “N-way K-shot” approach. Here, $N$ refers to the number of active labels (i.e., the number of singleton LC-classes in an episode) and $K$ represents the number of items supporting each singleton LC-class. To prevent oversampling, an item sampled for a singleton class is also counted for any other active classes it belongs to, ensuring that the number of items for each class stays close to $K$ .
Given the support items $(x_i,y_i)∈ S$ , the set of all LC-classes $L$ is computed as:
$$
L=\bigcup_(x_i,\mathbf{y_i)∈ S}P(y_i), \tag{5}
$$
where $P(y_i)$ is the power set of the labels of the $i$ -th support item, excluding the empty set. The total number of LC-classes is given by the cardinality $|L|$ of the computed set The scalability issues of the method in terms of LC-classes are being discussed in detail in Section 5 - 5.3..
We denote an LC-class as $L_j$ , where $j=1,2,...,|L|$ . For each LC-class, one or more support items include it in the power set of their labels, forming the set $S_j⊆ S$ , defined as: $S_j=\{(x_i,y_i)∈ S\mid L_j∈P(y _i)\}.$
The LC-Protonet representation $p_j$ for the corresponding class is computed by averaging the embeddings of the items in $S_j$ :
$$
p_j=\frac{1}{|S_j|}∑_(x_i,\mathbf{y_i)∈ S_j
}f_θ(x_i), \tag{6}
$$
where $f_θ$ is the embedding mapping model with $θ$ trainable parameters.
In each episode, a specified number of query items for each active label is sampled to form the query set $Q$ . Given a query item $x_i$ with an initial label vector $y_i∈\{0,1\}^N$ (a multi-hot vector of dimension $N$ ), the expanded label vector $y_mH_{i}∈\{0,1\}^|L|$ is constructed by assigning a value of 1 to each of the item’s LC-classes: $y_mH_{i}(j)=1 if L_j∈P(y_i ), ∀ j=1,2,...,|L|.$ The loss function is based on the binary cross-entropy:
$$
\begin{split}Loss(x_i,&y_mH_{i},p)=-y_mH_{i}\log(σ(-d(f_θ(x_i),p)))+\\
&(1-y_mH_{i})\log(1-σ(-d(f_θ(x_i),p
))),\end{split} \tag{7}
$$
where $d$ is the distance function, $σ$ the sigmoid function and $p$ the LC-Protonets representations. We minimize the loss for all items in the query set $Q$ .
Inference phase: At the inference phase, the support set $S$ is created following a similar “N-way K-shot” setup used during training. The LC-classes $L$ and their corresponding LC-Protonet representations $p_j$ (for $j=1,2,...,|L|$ ) are computed. For a query instance, the distances to all LC-Protonets are calculated, and the instance is assigned to the LC-class represented by the nearest LC-Protonet:
$$
\hat{y_i}=\arg\min_L_{j∈ L}d(f_θ(x_i),\mathbf
{p}_j). \tag{8}
$$
It is important to note that the training phase of LC-Protonets closely follows the extension of Prototypical Networks [8] for the multi-label setting. However, in the latter, only singleton LC-classes are considered in $L$ , making it a special case of the LC-Protonets approach. Another difference lies in the inference process, where the probabilities after a Sigmoid layer have to be utilized for classification as opposed to the direct approach adopted by the proposed method. LC-Protonets transforms the multi-label task to a single-label problem, where every combination of labels $L_j$ is a descriptive label.
## 4 EXPERIMENTS
Table 1: Number of recordings, total tags, and the relative frequency of the $i$ th most frequent (and last well-represented) tag for each dataset.
| MagnaTagATune | 25863 | 188 | 50 th: 1.89% |
| --- | --- | --- | --- |
| FMA-medium | 25000 | 151 | 20 th: 2.68% |
| Lyra | 1570 | 146 | 30 th: 4.78% |
| Turkish-makam | 5297 | 217 | 30 th: 2.83% |
| Hindustani | 1204 | 273 | 20 th: 2.49% |
| Carnatic | 2612 | 283 | 20 th: 2.03% |
Table 2: Macro-F1 (M-F1) and micro-F1 (m-F1) scores (%) with standard deviation, when aggregating the performance of the three methods over all datasets and training setups, for various “N-way” ML-FSL tasks, with a “3-shot” approach applied to all of them.
| ML-FSL task | 5-way 3-shot | 15-way 3-shot | | | | | | |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| classes type | Base | Novel | Base | Novel | | | | |
| method / metric | M-F1 | m-F1 | M-F1 | m-F1 | M-F1 | m-F1 | M-F1 | m-F1 |
| ML-PNs | 65.21±3.48 | 66.12±3.55 | 46.32±3.12 | 45.92±2.85 | 39.31±1.34 | 44.23±1.7 | 21.45±1.05 | 21.02±0.99 |
| One-vs.-Rest | 64.69±3.37 | 65.84±3.35 | 42.69±2.85 | 42.5±2.74 | 35.44±1.32 | 39.4±1.44 | 18.8±1.17 | 18.56±1.13 |
| LC-Protonets (ours) | 62.6±3.97 | 66.26±3.92 | 47.89±5.39 | 49.34±4.98 | 42.84±2.19 | 56.28±2.3 | 28.5±2.91 | 31.37±3.02 |
| ML-FSL task | 30-way 3-shot | 45-way 3-shot | 60-way 3-shot | | | | | |
| classes type | Base | Base & Novel | Base & Novel | Base & Novel | | | | |
| method / metric | M-F1 | m-F1 | M-F1 | m-F1 | M-F1 | m-F1 | M-F1 | m-F1 |
| ML-PNs | 29.84±0.92 | 32.57±0.97 | 24.61±0.66 | 29.26±1.02 | 19.74±0.53 | 23.05±0.82 | 17.49±0.49 | 19.45±0.54 |
| One-vs.-Rest | 25.64±0.97 | 27.81±1.09 | 21.74±0.79 | 25.36±1.04 | 17.06±0.61 | 19.58±0.8 | 14.79±0.55 | 16.17±0.69 |
| LC-Protonets | 36.77±1.97 | 50.77±1.44 | 31.31±1.73 | 52.65±1.57 | 28.09±1.42 | 50.28±1.65 | 28.45±1.51 | 46.48±1.47 |
### 4.1 DATASETS AND METRICS
We incorporate a range of datasets from both mainstream and world music traditions for our study. For Western music, we use the MagnaTagATune dataset [47], a standard for auto-tagging, and the medium version of FMA [48]. For world music, we utilize the Lyra dataset [49], a collection of Greek folk music, along with three datasets from the CompMusic Corpora [3]: the Turkish-makam corpus [50, 51], which includes recordings from the Turkish tradition, and two datasets of Indian art music, Hindustani and Carnatic [52].
Table 1 provides details on the number of recordings, total tags, and the relative frequency of the last well-represented label for each dataset. We consider as well-represented the $i$ most frequent tags for each dataset based on their successful inclusion in the supervised learning approach from [11]: $50$ for MagnaTagATune, $30$ for Lyra and Turkish-makam, and $20$ for FMA-medium, Hindustani, and Carnatic. The data preparation for the automatic audio tagging task followed the same process described in [11]. To use these datasets for few-shot learning, we split the labels for training and testing, and we provide those splits to the public repository for reproducibility.
For evaluation, we selected Macro-F1 and Micro-F1 scores. Since our method directly predicts a set of labels without assigning probabilities to individual labels, calculating metrics like Area Under the Curve (AUC) is not straightforward. F1 score is the harmonic mean of the precision and recall scores and in its Macro- setting, the mean of the individual label scores is calculated. Micro-F1 computes metrics globally by aggregating true positives, false negatives, and false positives across all samples.
### 4.2 BACKBONE MODEL
In few-shot learning, each sample is embedded into a feature space by a backbone model. Given the VGG-ish [53] model’s ease of training and proven effectiveness in both supervised learning [11, 6] and ML-FSL tasks [14], we selected it as our backbone model. The architecture consists of a 7-layer Convolutional Neural Network (CNN) with $3× 3$ convolution filters and $2× 2$ max-pooling layers, followed by a couple of fully-connected layers. The model processes log mel-spectrograms as input features.
### 4.3 COMPARATIVE APPROACHES
ML-PNs: Our method is compared to the extension of Prototypical Networks for multi-label classification, referred to as “ML-PNs”. As described in Section 3 - 3.1, this approach uses a Sigmoid layer instead of Softmax for classification, and binary cross-entropy loss instead of categorical cross-entropy during training, compared to the standard single-label Prototypical Networks [8].
One-vs.-Rest: Another comparative approach is the “One-vs.-Rest” strategy introduced in [46]. In this method, the support set is divided into several subsets during training, where each subset focuses on a query’s label along with $N-1$ other classes in an “N-way K-shot” format. The goal is to decompose the multi-label problem into multiple binary classification tasks.
<details>
<summary>extracted/5860914/figs/vis_magnatagatune_12_way_5_shot.png Details</summary>
### Visual Description
## Scatter Plot: Musical Genre and Instrument Distribution
### Overview
The image displays two side-by-side scatter plots visualizing the distribution of musical genres and instruments across tracks. Each plot contains clusters of colored circles representing specific musical elements, with a shared legend in the center. The left panel shows sparse colored circles, while the right panel exhibits denser, overlapping clusters. Gray circles represent background noise or unclassified elements.
### Components/Axes
- **Legend**: Located centrally between the two panels, listing 12 categories with color codes:
- Female vocals (blue)
- Hard rock (orange)
- Techno (dark blue)
- Guitar (red)
- Drums (purple)
- Slow (brown)
- Male vocals (pink)
- Jazz (dark red)
- Classical (yellow)
- Electro (cyan)
- Drum (green)
- Strings (olive green)
- **Panels**:
- Left panel: Sparse colored circles (12-15 visible)
- Right panel: Dense clusters (20-25 visible)
- **Background**: White with gray circles representing unclassified elements.
### Detailed Analysis
1. **Left Panel**:
- **Female vocals** (blue): Single circle in lower-right quadrant.
- **Hard rock** (orange): Single circle in lower-right quadrant.
- **Techno** (dark blue): Single circle in upper-middle region.
- **Guitar** (red): Single circle in lower-middle region.
- **Drums** (purple): Single circle in upper-middle region.
- **Slow** (brown): Single circle in upper-left quadrant.
- **Male vocals** (pink): Single circle in lower-left quadrant.
- **Jazz** (dark red): Single circle in upper-middle region.
- **Classical** (yellow): Single circle in lower-left quadrant.
- **Electro** (cyan): Single circle in upper-right quadrant.
- **Drum** (green): Single circle in lower-middle region.
- **Strings** (olive green): Single circle in lower-left quadrant.
- Gray circles: ~50 scattered across all quadrants.
2. **Right Panel**:
- **Female vocals** (blue): 3-4 circles clustered in upper-right quadrant.
- **Hard rock** (orange): 2 circles overlapping with red (guitar) in lower-right quadrant.
- **Techno** (dark blue): 2 circles overlapping with purple (drums) in upper-middle region.
- **Guitar** (red): 3 circles overlapping with orange (hard rock) in lower-right quadrant.
- **Drums** (purple): 4 circles overlapping with dark blue (techno) in upper-middle region.
- **Slow** (brown): 2 circles in upper-left quadrant.
- **Male vocals** (pink): 3 circles overlapping with red (guitar) in lower-middle region.
- **Jazz** (dark red): 3 circles in upper-middle region.
- **Classical** (yellow): 2 circles in lower-left quadrant.
- **Electro** (cyan): 2 circles in upper-right quadrant.
- **Drum** (green): 3 circles overlapping with purple (drums) in upper-middle region.
- **Strings** (olive green): 2 circles in lower-left quadrant.
- Gray circles: ~70 scattered across all quadrants.
### Key Observations
1. **Density Differences**: The right panel contains 3-4x more colored circles than the left, suggesting a focus on complex compositions.
2. **Overlap Patterns**:
- Orange (hard rock) and red (guitar) frequently overlap in the lower-right quadrant.
- Purple (drums) and dark blue (techno) overlap in the upper-middle region.
- Pink (male vocals) and red (guitar) overlap in the lower-middle region.
3. **Spatial Distribution**:
- Lower-left quadrant: Classical (yellow), male vocals (pink), strings (
</details>
Figure 2: t-SNE visualization of query items (in grey) and prototype embeddings (in distinct colors) for a “12-way 5-shot” ML-FSL task on the MagnaTagATune dataset. The left panel shows prototypes generated by the “ML-PNs” method (one per class), while the right panel displays those formed using the “LC-Protonets” method, where different colors within each prototype indicate the specific label combination it represents.
### 4.4 EXPERIMENTAL SETUP
We split the labels of each dataset into training and testing sets. The training set is used both during the training phase of ML-FSL models and for pre-training the backbone VGG-ish model via supervised learning, while the testing set is used to form the novel classes for evaluation. For the audio recordings, we use the same splits for training, validation, and testing as in previous studies [11, 6].
To train the ML-FSL models, we employ three setups: (i) training from scratch with random weight initialization, (ii) full fine-tuning of a pre-trained backbone model, and (iii) fine-tuning only the last layer of the pre-trained backbone model. The model architecture remains the same across all setups, except that in the two fine-tuning cases, a VGG-ish model pre-trained on the well-represented tags is transferred as the backbone, with only the final classification layer excluded.
In few-shot learning, base classes are those seen during training, while novel classes are unseen. We evaluate ML-FSL models on “Base”, “Novel” and “Base & Novel” classes, with the latter including an equal mix of unseen and seen tags. This allows us to assess how well the model handles both seen and unseen classes during inference. Various values of $N$ (the number of classes) are tested, while $K$ is kept constant (3-shot) across the “N-way K-shot” ML-FSL tasks. Importantly, the same base classes used for training an ML-FSL model from scratch are also used for pre-training the backbone model via supervised learning, ensuring that the novel classes remain unseen for models with a pre-trained backbone.
Cosine distance is used as the distance metric, and all methods are trained using episodic learning with a $12$ -way $3$ -shot setup. This setup is chosen to accommodate the low-resource nature of music data, as the absolute number of labels differs across domains. For instance, Hindustani, Carnatic, and FMA-medium datasets have only 20 labels in the training split. Additionally, selecting 3 examples per label allows under-represented labels to be included. $50$ episodes are sampled for each epoch and $3$ query items per label are utilized to compute the loss in each one of them.
The validation set is formed by holding out 5 classes from the training set during learning. The Adam optimizer [54] is used, and early stopping is applied based on the Macro-F1 score on the validation set at the end of each epoch. Regarding the input, a 512-point FFT with a 50% overlap is applied to the audio signal, and the Mel bands are set to 128. During training, a random chunk of each audio recording is selected, while during testing, the average embedding of all chunks forms the representation of an instance.
### 4.5 TWO-STEP LEARNING METHOD
In imbalanced datasets, it is common to encounter a large number of labels that occur infrequently, leading to a long-tailed label distribution. When training models using supervised learning, a threshold is often set to include only the most frequent categories. To address this limitation, we propose a two-step method that combines supervised and few-shot learning. Unlike the ML-FSL training setups that use a pre-trained backbone model, no fine-tuning is applied in this approach.
As illustrated in Figure 3, the first step involves training a deep learning model on well-represented tags using supervised learning. The model is optimized with a Sigmoid classification layer and binary cross-entropy loss. In the second step, the pre-trained model is frozen and used as a backbone to map data samples into an embedding space defined by its penultimate layer. We extend the tag set and perform inference on any query item by applying LC-Protonets on top of the pre-trained model. Without additional training, LC-Protonets can classify previously unseen, under-represented labels, including those from the long tail of world music datasets, using just a few examples per label. In our experiments, we use $3$ examples per label.
<details>
<summary>extracted/5860914/figs/two_step_architecture.png Details</summary>
### Visual Description
## Diagram: Two-Step Machine Learning Process for Classification
### Overview
The diagram illustrates a two-phase machine learning workflow for classification tasks. The first phase uses supervised learning with a pre-trained VGG-ish model to generate binary tags (1/0). The second phase introduces few-shot learning capabilities through LC-Protonets, enabling adaptation to novel classes with limited examples. The process includes feedback loops for continuous improvement.
### Components/Axes
1. **Input**: Red spectrogram-like images (likely audio/text features)
2. **Phase 1 Components**:
- **VGG-ish**: Pre-trained convolutional neural network
- **Sigmoid**: Binary classification output (1 = well-represented, 0 = not)
3. **Phase 2 Components**:
- **Pre-trained VGG-ish**: Base model from Phase 1
- **LC-Protonets**: Meta-learning component with colored nodes (yellow, orange, blue, purple)
- **Extended Tags**: Binary outputs (1/0) for novel classes
4. **Support System**:
- **Support Items**: 4 red images (50% of total support data)
- **Novel Classes**: 4 red images (50% of total novel data)
5. **Feedback Loop**: Dashed arrow connecting Phase 2 outputs back to VGG-ish
### Detailed Analysis
- **Phase 1 Flow**:
Input → VGG-ish → Sigmoid → Binary Tags
- Sigmoid outputs show clear binary separation (1/0)
- No uncertainty values provided for Phase 1 predictions
- **Phase 2 Flow**:
Input → Pre-trained VGG-ish → LC-Protonets → Extended Tags
- LC-Protonets contains 5 colored nodes (4 distinct colors + gray)
- Color coding purpose unspecified in diagram
- Feedback loop connects extended tags back to base model
- **Support/Novel Distribution**:
- Equal 50/50 split between support items and novel classes
- All support/novel examples use identical red spectrogram format
- No quantitative performance metrics shown for few-shot adaptation
### Key Observations
1. **Model Architecture**:
- VGG-ish serves as foundational model in both phases
- LC-Protonets introduces meta-learning capabilities without modifying base architecture
2. **Data Flow**:
- Phase 2 maintains identical input format to Phase 1
- Feedback loop suggests iterative model refinement
3. **Color Coding**:
- LC-Protonets uses 4 distinct colors (yellow, orange, blue, purple) + gray
- No legend explaining color significance
- Binary outputs maintain consistent 1/0 labeling across phases
### Interpretation
This diagram demonstrates a hybrid approach combining:
1. **Supervised Learning Foundation**: Using VGG-ish for initial feature extraction and binary classification
2. **Few-Shot Adaptation**: LC-Protonets enables rapid adaptation to new classes with minimal examples
3. **Continuous Learning**: Feedback loop from novel class performance back to base model suggests online learning capabilities
The equal split between support and novel classes (4 each) implies a balanced few-shot scenario, though real-world applications might require different distributions. The absence of uncertainty quantification in both phases' outputs is a notable limitation. The color-coded LC-Protonets component likely represents different meta-learning strategies or class prototypes, but without explicit legend information, this remains speculative.
The process appears designed for scenarios where:
- Initial classification requires high accuracy (Phase 1)
- New classes emerge frequently with limited examples (Phase 2)
- Model must continuously improve through real-world feedback
</details>
Figure 3: Depiction of the two-step learning method: in the first step, a model is trained on well-represented tags via supervised learning; in the second step, the tag set is extended by applying the LC-Protonets method on the previously trained model, which serves as the backbone.
Table 3: Macro-F1 (M-F1) and micro-F1 (m-F1) scores (%) for a “30-way 3-shot” task with “Base & Novel” classes, under four conditions: (i) training from scratch, (ii) full fine-tuning of a pre-trained backbone model, (iii) fine-tuning of the final layer of a pre-trained backbone, and (iv) using a pre-trained backbone without any fine-tuning. Rows represent the multi-label few-shot learning methods, while columns correspond to the different datasets.
| ML-PNs One-vs.-Rest LC-Protonets | 19.89±0.58 16.72±0.69 21.58±1.26 | 21.33±0.83 17.36±0.91 29.49±1.71 | 18.32±0.44 13.73±0.67 18.91±1.24 | 20.04±0.48 14.82±0.41 34.75±1.4 | 31.4±1.03 29.05±0.33 39.47±3.68 | 37.5±1.38 33.9±1.07 59.82±2.22 | 20.29±0.09 20.3±0.12 21.52±2.38 | 22.12±0.1 22.12±0.11 37.41±2.16 | 18.16±0.55 17.86±0.45 24.71±3.89 | 23.89±1.46 22.77±1.38 50.82±2.81 | 20.12±0.74 20.23±1.23 17.96±0.38 | 30.11±1.12 27.64±0.86 54.63±1.13 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| method | pre-trained backbone and full fine-tuning | | | | | | | | | | | |
| ML-PNs | 25.0±0.28 | 27.63±0.46 | 22.08±0.32 | 23.71±0.54 | 35.64±1.05 | 41.72±1.22 | 32.19±1.15 | 32.72±1.17 | 23.1±0.43 | 30.4±0.74 | 22.04±0.64 | 31.41±1.92 |
| One-vs.-Rest | 19.82±0.88 | 20.96±1.16 | 18.7±0.38 | 19.64±0.7 | 29.02±0.2 | 33.36±0.35 | 26.05±1.14 | 28.08±1.31 | 18.36±0.75 | 23.52±1.33 | 20.72±0.35 | 28.45±0.78 |
| LC-Protonets | 33.66±1.14 | 43.37±1.89 | 33.37±0.79 | 48.83±1.1 | 45.29±1.95 | 65.99±1.01 | 38.59±1.98 | 57.31±1.75 | 35.07±2.12 | 59.03±1.88 | 23.16±1.31 | 63.69±2.07 |
| method | pre-trained backbone and fine-tuning of the last layer | | | | | | | | | | | |
| ML-PNs | 24.45±0.48 | 26.94±0.73 | 20.88±0.3 | 22.65±0.37 | 36.72±0.74 | 43.16±0.77 | 28.52±2.15 | 30.73±1.83 | 22.84±0.34 | 31.16±1.15 | 21.38±0.62 | 29.46±2.12 |
| One-vs.-Rest | 19.86±1.86 | 20.82±1.91 | 19.23±0.35 | 20.52±0.52 | 31.75±0.95 | 37.03±1.33 | 28.95±1.76 | 30.77±1.52 | 20.18±1.44 | 26.29±1.95 | 20.73±0.65 | 28.44±1.09 |
| LC-Protonets | 33.5±1.07 | 43.27±1.6 | 33.04±1.39 | 48.68±1.56 | 47.31±2.54 | 68.58±0.82 | 38.52±1.67 | 57.99±1.19 | 34.64±1.71 | 60.04±1.16 | 23.25±0.55 | 63.92±0.85 |
| method | pre-trained backbone without any fine-tuning | | | | | | | | | | | |
| ML-PNs | 13.62±0.01 | 14.3±0.01 | 10.58±0.0 | 11.18±0.0 | 28.93±0.08 | 33.25±0.09 | 20.28±0.08 | 22.11±0.09 | 17.49±0.09 | 21.83±0.12 | 20.88±0.06 | 27.76±0.04 |
| One-vs.-Rest | 13.62±0.01 | 14.31±0.01 | 10.58±0.01 | 11.19±0.01 | 28.9±0.08 | 33.22±0.1 | 20.25±0.05 | 22.07±0.04 | 17.6±0.1 | 21.94±0.14 | 20.91±0.04 | 27.8±0.06 |
| LC-Protonets | 33.52±0.96 | 43.24±1.61 | 33.73±1.02 | 49.2±1.51 | 47.32±3.03 | 68.95±1.61 | 37.23±1.38 | 56.83±0.67 | 35.09±2.28 | 59.69±1.46 | 22.42±0.63 | 63.66±1.06 |
## 5 RESULTS
### 5.1 ML-FSL TASKS
In Table 2, the aggregated results of the LC-Protonets method and the two comparative approaches are presented. These results were calculated by averaging performance across all datasets and training setups: from scratch, full fine-tuning, and fine-tuning of the last layer. Each experiment was run five times, and the standard deviation is reported. We present both macro-F1 and micro-F1 scores for different numbers of labels, ranging from $5$ to $60$ . The evaluations were performed on “Base” and “Novel” classes for smaller numbers of classes, and on “Base & Novel” classes for larger numbers.
LC-Protonets outperformed other methods in nearly all tasks, except for the $5$ -way $3$ -shot task with base classes, where ML-PNs performed better in terms of macro-F1 score. In the $15$ -way $3$ -shot task, LC-Protonets showed superior performance on base classes and widened this gap further when novel classes were used. As the number of classes increased, LC-Protonets demonstrated substantially better performance compared to the other approaches.
In terms of standard deviation, we observed larger values for fewer-shot conditions, such as the $5$ -way task, due to the random sampling of a small number of active classes in imbalanced datasets. Additionally, LC-Protonets’ reliance on support set sampling for deriving LC-classes leads to higher standard deviation values compared to the other methods.
Figure 2 highlights the differences between the prototype representations generated by ML-PNs and LC-Protonets. While ML-PNs create one prototype per class, LC-Protonets populate the embedding space with representations derived from the power sets of the support item labels. We believe this enhanced positive sampling of the feature space contributes to the significant performance improvement seen in the results.
Table 3 presents the results of the $30$ -way $3$ -shot task on “Base & Novel” classes for each dataset and training setup. When training from scratch, the LC-Protonets method showed improvement in all cases except for the macro-F1 evaluation on the Carnatic dataset, where One-vs.-Rest performed better. The difference between LC-Protonets and the comparative approaches was more evident in the micro-F1 scores, as also noted in the aggregated results.
When using a pre-trained backbone model followed by full fine-tuning with episodic learning, the performance of all methods significantly improved across all datasets. However, the gap between LC-Protonets and the comparative approaches widened further. For instance, in MagnaTagATune, ML-PNs improved from $19.89\$ to $25.0\$ , while LC-Protonets increased from $21.58\$ to $33.66\$ in macro-F1 score. When only the last layer of the pre-trained model was fine-tuned, performance remained similar across datasets, with the exception of Lyra, where this approach led to slightly better results.
Finally, the last three rows of Table 3 report the performance when a pre-trained backbone model was used without any fine-tuning. Interestingly, LC-Protonets maintained performance levels similar to those seen in the fine-tuning setups. This suggests that the method relies more on the quality of the representations provided by the backbone model than on episodic learning. By contrast, the performance of the comparative approaches dropped significantly, as they encountered challenges with multi-label classification without training, often assigning all labels to all samples. This can be seen in the zero standard deviation of ML-PNs on the FMA-medium dataset, for example.
<details>
<summary>extracted/5860914/figs/scalability.png Details</summary>
### Visual Description
## Line Graph: Relationship Between N-labels and LC-Protons/Time
### Overview
The image is a dual-axis line graph comparing two metrics: the number of LC-Protons and the time (in milliseconds) per query-item, both plotted against the number of N-labels. The graph uses logarithmic scales for both y-axes to accommodate exponential growth trends. Two data series are represented: an orange line for LC-Protons and a blue line for time.
### Components/Axes
- **X-axis (Horizontal)**: Labeled "N-labels" with values ranging from 10 to 60 in increments of 10.
- **Left Y-axis**: Labeled "# LC-Protons" with a logarithmic scale (10¹ to 10⁵).
- **Right Y-axis**: Labeled "time (msec) / query-item vs N-labels" with a logarithmic scale (10¹ to 10⁵).
- **Legend**: Located in the top-left corner, with:
- **Orange line**: "number of LCPs vs N-labels"
- **Blue line**: "time (msec) / query-item vs N-labels"
### Detailed Analysis
#### Orange Line (LC-Protons)
- **Data Points**:
- N=10: 21 LC-Protons
- N=15: 644 LC-Protons
- N=30: 3,787 LC-Protons
- N=45: 13,233 LC-Protons
- N=60: 38,953 LC-Protons
- **Trend**: Exponential growth. The line slopes steeply upward, with values increasing by orders of magnitude as N-labels rise.
#### Blue Line (Time)
- **Data Points**:
- N=10: 16ms
- N=15: 31ms
- N=30: 118ms
- N=45: 359ms
- N=60: 912ms
- **Trend**: Linear growth. The line slopes gradually upward, with time increasing proportionally to N-labels.
### Key Observations
1. **Exponential vs. Linear Growth**: The orange line (LC-Protons) grows exponentially, while the blue line (time) grows linearly.
2. **Divergence at Higher N-labels**: At N=60, LC-Protons reach 38,953 (10⁴.⁶), while time is 912ms (10².⁹⁶), highlighting a stark disparity in scaling.
3. **Consistent Scaling**: Both metrics increase monotonically with N-labels, but LC-Protons dominate in magnitude.
### Interpretation
The graph demonstrates that as the number of N-labels increases, the system's LC-Proton count grows exponentially, suggesting a potential bottleneck or inefficiency in resource allocation. Meanwhile, the time per query-item increases linearly, indicating that computational overhead scales predictably but less severely. This divergence implies that optimizing LC-Proton usage could be critical for maintaining performance at scale, as the exponential growth may lead to resource exhaustion long before time-based constraints become limiting.
</details>
Figure 4: Scalability metrics of the proposed method, averaged across all datasets. The $x$ -axis represents the number of labels, the left $y$ -axis shows the number of LC-Protonets, and the right $y$ -axis indicates the inference time per item. Both $y$ -axes use the same logarithmic scale.
Table 4: Macro-F1 (M-F1) and micro-F1 (m-F1) scores (%) for the “VGG-ish” model on the set of well-represented tags, and for the two-step method “VGG-ish & LCPs” (VGG-ish & LC-Protonets) on both the well-represented and extended tag sets for each dataset.
| dataset # tags: original / extended method / metric | MagnaTagATune 50 M-F1 | FMA-medium 80 m-F1 | Lyra 20 M-F1 | 40-m-F1 | 30 M-F1 | 60-m-F1 | M-F1 | m-F1 | M-F1 | m-F1 | M-F1 | m-F1 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| VGG-ish | 26.74±0.63 | 42.29±0.58 | - | - | 36.90±0.76 | 59.61±0.84 | - | - | 30.48±1.23 | 67.14±1.17 | - | - |
| VGG-ish & LCPs | 33.09±0.67 | 39.28±1.42 | 26.4±0.1 | 37.31±0.19 | 40.94±1.61 | 53.51±0.59 | 29.12±1.16 | 45.37±1.38 | 47.32±3.03 | 68.95±1.61 | 46.05±2.26 | 69.03±1.78 |
| dataset | Turkish-makam | Hindustani | Carnatic | | | | | | | | | |
| # tags: original / extended | 30 | 60 | 20 | 35 | 20 | 40 | | | | | | |
| method / metric | M-F1 | m-F1 | M-F1 | m-F1 | M-F1 | m-F1 | M-F1 | m-F1 | M-F1 | m-F1 | M-F1 | m-F1 |
| VGG-ish | 44.95±0.82 | 79.11±0.98 | - | - | 46.07±1.12 | 76.60±1.29 | - | - | 35.49±1.54 | 84.82±1.71 | - | - |
| VGG-ish & LCPs | 37.23±1.38 | 56.83±0.67 | 30.07±1.31 | 56.22±1.14 | 40.69±1.47 | 64.38±0.97 | 31.33±1.62 | 58.38±1.94 | 32.1±1.18 | 64.84±1.22 | 18.13±0.48 | 64.25±0.66 |
### 5.2 TWO-STEP LEARNING METHOD
The results of the proposed two-step learning method are shown in Table 4. For each dataset, two different numbers of tags are used. The smaller number, such as $20$ for FMA-medium, corresponds to the well-represented tags on which the VGG-ish model was trained, while the larger number, $40$ , represents the extended set of tags. The performance of both the “VGG-ish” model and the “VGG-ish & LCPs” (VGG-ish & LC-Protonets) method on the well-represented tags is reported, and for the latter, its performance on the extended tag set is also included.
When examining the macro-F1 scores for both methods on the smaller set of tags, we observe similar performance across most datasets. The key architectural difference between the two approaches is the replacement of the VGG-ish Sigmoid classification layer with the LC-Protonets framework, which classifies an unknown sample to the label combination represented by the nearest LCP. The utilization of the LCPs offers a straightforward way to expand the number of labels. For instance, the tags in MagnaTagATune can be extended from $50$ to $80$ , in Hindustani from $20$ to $35$ , and doubled for the other datasets.
The percentage drop in macro-F1 performance is significantly smaller than the percentage increase in the number of tags. For example, in the Turkish-makam dataset, the macro-F1 score drops from $37.23\$ to $30.07\$ as the number of tags increases from $30$ to $60$ . An exception is the Lyra dataset, where performance on the extended tag set remains nearly identical to the well-represented tags. This could be due to stronger correlations between tags in Lyra compared to the other datasets.
### 5.3 SCALABILITY
As the LC-classes are derived from the power sets of the sample labels, the number of LC-Protonets increases significantly as the number of classes $N$ grows. This results in a corresponding increase in inference time, as the distances from all LC-Protonets must be computed for each query item. In Figure 4, the number of classes $N$ is shown on the $x$ -axis, while the left $y$ -axis represents the number of LC-Protonets, and the right $y$ -axis shows the inference time per query item (in milliseconds), averaged across all datasets.
When $N$ increases from $5$ to $15$ , the number of LC-Protonets rises by a factor of 30, from $21$ to $644$ , while inference time increases from $16$ to $31$ milliseconds. As the number of classes continues to grow, the number of LC-Protonets increases substantially, reaching $38,953$ for $60$ tags, and the inference time also rises to $912$ milliseconds. However, the gradient of the increase in inference time is less steep compared to the growth in LC-Protonets. We believe these scalability issues can be mitigated by incorporating optimization techniques, such as skipping the computation of distances from LC-Protonets with identical representations that correspond to different LC-classes (which may be hierarchically related). We plan to explore these methods in future work.
It is important to note that the inference process only runs during testing and not during model training. The average training time across all three methods was less than an hour, with no significant differences between them. Finally, the trainable parameters amount to $3.66$ million when training from scratch or performing full fine-tuning, and $262,000$ when only fine-tuning the last layer. The experiments were conducted on an NVIDIA RTX A5000 GPU.
## 6 CONCLUSIONS
In this paper, we introduced LC-Protonets, a method for multi-label few-shot classification, and evaluated its performance on automatic audio tagging across diverse music datasets. LC-Protonets consistently outperformed comparative approaches in different domains and ML-FSL tasks, using various training setups. We also explored the impact of fine-tuning a pre-trained backbone model. The results show that utilizing a pre-trained model significantly benefits all ML-FSL methods, and experimenting with both full fine-tuning and fine-tuning of the last layer is recommended.
Additionally, we showed that LC-Protonets can perform well using pre-trained model embeddings without any fine-tuning, unlike the comparative approaches. This led to the development of a two-step learning method, where LC-Protonets can successfully expand the tag set of a dataset by leveraging a model pre-trained on well-represented tags.
However, some limitations remain. The scalability of LC-Protonets, while advantageous in performance, leads to increased inference time as the number of labels grows. This could affect the efficiency of the method when applied to datasets with numerous classes. Additionally, the method’s reliance on support set sampling introduces variability, which may impact performance. In future work, we aim on addressing these challenges to further improve robustness and scalability. We will also explore using different architectures and pre-trained models as backbones and attempt experiments in other modalities.
We hope that this work can motivate the development of machine learning methods that can lead to the exploration and understanding of under-represented domains, such as world music cultures, by unlocking the data scarcity problem with a simple yet effective way.
## References
- [1] B. M. Lake, R. Salakhutdinov, and J. B. Tenenbaum, “Human-level concept learning through probabilistic program induction,” Science, vol. 350, no. 6266, pp. 1332–1338, 2015.
- [2] Y. Wang, Q. Yao, J. T. Kwok, and L. M. Ni, “Generalizing from a few examples: A survey on few-shot learning,” ACM Comput. Surv., vol. 53, no. 3, pp. 63:1–63:34, 2021.
- [3] X. Serra, “Creating research corpora for the computational study of music: the case of the compmusic project,” in Semantic Audio. Audio Engineering Society, 2014.
- [4] K. Choi, “Deep neural networks for music tagging,” Ph.D. dissertation, Queen Mary University of London, UK, 2018.
- [5] T. Kim, J. Lee, and J. Nam, “Sample-level CNN architectures for music auto-tagging using raw waveforms,” in ICASSP. IEEE, 2018, pp. 366–370.
- [6] M. Won, A. Ferraro, D. Bogdanov, and X. Serra, “Evaluation of cnn-based automatic music tagging models,” CoRR, vol. abs/2006.00751, 2020.
- [7] J. Lee, J. Park, K. L. Kim, and J. Nam, “Sample-level deep convolutional neural networks for music auto-tagging using raw waveforms,” CoRR, vol. abs/1703.01789, 2017.
- [8] J. Snell, K. Swersky, and R. S. Zemel, “Prototypical networks for few-shot learning,” in NIPS, 2017, pp. 4077–4087.
- [9] O. Vinyals, C. Blundell, T. Lillicrap, K. Kavukcuoglu, and D. Wierstra, “Matching networks for one shot learning,” in NIPS, 2016, pp. 3630–3638.
- [10] F. Sung, Y. Yang, L. Zhang, T. Xiang, P. H. S. Torr, and T. M. Hospedales, “Learning to compare: Relation network for few-shot learning,” in CVPR. Computer Vision Foundation / IEEE Computer Society, 2018, pp. 1199–1208.
- [11] C. Papaioannou, E. Benetos, and A. Potamianos, “From west to east: Who can understand the music of the others better?” in ISMIR, 2023, pp. 311–318.
- [12] K. Choi, G. Fazekas, M. B. Sandler, and K. Cho, “Transfer learning for music classification and regression tasks,” in ISMIR, 2017, pp. 141–149.
- [13] A. van den Oord, S. Dieleman, and B. Schrauwen, “Transfer learning by supervised pre-training for audio-based music classification,” in ISMIR, 2014, pp. 29–34.
- [14] J. Liang, H. Phan, and E. Benetos, “Learning from taxonomy: Multi-label few-shot classification for everyday sound recognition,” in ICASSP. IEEE, 2024, pp. 771–775.
- [15] Y. Ma, A. Øland, A. Ragni, B. M. Del Sette, C. Saitis, C. Donahue, C. Lin, C. Plachouras, E. Benetos, E. Quinton et al., “Foundation models for music: A survey,” CoRR, vol. abs/2408.14340, 2024.
- [16] J. Spijkervet and J. A. Burgoyne, “Contrastive learning of musical representations,” in ISMIR, 2021, pp. 673–681.
- [17] L. Wang and A. van den Oord, “Multi-format contrastive learning of audio representations,” CoRR, vol. abs/2103.06508, 2021.
- [18] P. Alonso-Jiménez, X. Favory, H. Foroughmand, G. Bourdalas, X. Serra, T. Lidy, and D. Bogdanov, “Pre-training strategies using contrastive learning and playlist information for music classification and similarity,” in ICASSP. IEEE, 2023, pp. 1–5.
- [19] M. Won, J. Spijkervet, and K. Choi, “Music classification: Beyond supervised learning, towards real-world applications,” CoRR, vol. abs/2111.11636, 2021.
- [20] S. Hershey, S. Chaudhuri, D. P. W. Ellis, J. F. Gemmeke, A. Jansen, R. C. Moore, M. Plakal, D. Platt, R. A. Saurous, B. Seybold, M. Slaney, R. J. Weiss, and K. W. Wilson, “CNN architectures for large-scale audio classification,” in ICASSP. IEEE, 2017, pp. 131–135.
- [21] J. Pons and X. Serra, “musicnn: Pre-trained convolutional neural networks for music audio tagging,” CoRR, vol. abs/1909.06654, 2019.
- [22] J. Lee, J. Park, K. L. Kim, and J. Nam, “Samplecnn: End-to-end deep convolutional neural networks using very small filters for music classification,” Applied Sciences, vol. 8, no. 1, p. 150, 2018.
- [23] A. Pandey and D. Wang, “TCNN: temporal convolutional neural network for real-time speech enhancement in the time domain,” in ICASSP. IEEE, 2019, pp. 6875–6879.
- [24] Y. Gong, Y. Chung, and J. R. Glass, “AST: audio spectrogram transformer,” in Interspeech. ISCA, 2021, pp. 571–575.
- [25] M. Won, Y. Hung, and D. Le, “A foundation model for music informatics,” in ICASSP. IEEE, 2024, pp. 1226–1230.
- [26] W. Hsu, B. Bolte, Y. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,” IEEE ACM Trans. Audio Speech Lang. Process., vol. 29, pp. 3451–3460, 2021.
- [27] Y. Zhang, J. Qin, D. S. Park, W. Han, C. Chiu, R. Pang, Q. V. Le, and Y. Wu, “Pushing the limits of semi-supervised learning for automatic speech recognition,” CoRR, vol. abs/2010.10504, 2020.
- [28] M. Panteli, “Computational analysis of world music corpora,” Ph.D. dissertation, Queen Mary University of London, UK, 2018.
- [29] E. Demirel, B. Bozkurt, and X. Serra, “Automatic makam recognition using chroma features,” in 8th International Workshop on Folk Music Analysis, 2018, pp. 19–24.
- [30] K. K. Ganguli, S. Sentürk, and C. Guedes, “Critiquing task- versus goal-oriented approaches: A case for makam recognition,” in ISMIR, 2022, pp. 369–376.
- [31] A. K. Sharma, G. Aggarwal, S. Bhardwaj, P. Chakrabarti, T. Chakrabarti, J. H. Abawajy, S. Bhattacharyya, R. Mishra, A. Das, and H. Mahdin, “Classification of indian classical music with time-series matching deep learning approach,” IEEE Access, vol. 9, pp. 102 041–102 052, 2021.
- [32] B. Nikzat and R. C. Repetto, “KDC: an open corpus for computational research of dastgāhi music,” in ISMIR, 2022, pp. 321–328.
- [33] D. Han, R. C. Repetto, and D. Jeong, “Finding tori: Self-supervised learning for analyzing korean folk song,” in ISMIR, 2023, pp. 440–447.
- [34] G. Koch, R. Zemel, and R. Salakhutdinov, “Siamese neural networks for one-shot image recognition,” in ICML deep learning workshop, vol. 2, no. 1, 2015.
- [35] T. Munkhdalai and H. Yu, “Meta networks,” in ICML, ser. Proceedings of Machine Learning Research, vol. 70. PMLR, 2017, pp. 2554–2563.
- [36] V. Joshi, M. E. Peters, and M. Hopkins, “Extending a parser to distant domains using a few dozen partially annotated examples,” in ACL (1). Association for Computational Linguistics, 2018, pp. 1190–1199.
- [37] L. Kaiser, O. Nachum, A. Roy, and S. Bengio, “Learning to remember rare events,” in ICLR, 2017.
- [38] B. M. Lake, C. Lee, J. R. Glass, and J. B. Tenenbaum, “One-shot learning of generative speech concepts,” in CogSci, 2014.
- [39] S. Ö. Arik, J. Chen, K. Peng, W. Ping, and Y. Zhou, “Neural voice cloning with a few samples,” in NeurIPS, 2018, pp. 10 040–10 050.
- [40] Y. Wang, J. Salamon, M. Cartwright, N. J. Bryan, and J. P. Bello, “Few-shot drum transcription in polyphonic music,” in ISMIR, 2020, pp. 117–124.
- [41] Y. Wang, D. Stoller, R. M. Bittner, and J. P. Bello, “Few-shot musical source separation,” in ICASSP. IEEE, 2022, pp. 121–125.
- [42] H. F. García, A. Aguilar, E. Manilow, and B. Pardo, “Leveraging hierarchical structures for few-shot musical instrument recognition,” in ISMIR, 2021, pp. 220–228.
- [43] A. Alfassy, L. Karlinsky, A. Aides, J. Shtok, S. Harary, R. S. Feris, R. Giryes, and A. M. Bronstein, “Laso: Label-set operations networks for multi-label few-shot learning,” in CVPR. Computer Vision Foundation / IEEE, 2019, pp. 6548–6557.
- [44] C. Simon, P. Koniusz, and M. Harandi, “Meta-learning for multi-label few-shot classification,” in WACV. IEEE, 2022, pp. 346–355.
- [45] M. Hu, S. Zhao, H. Guo, C. Xue, H. Gao, T. Gao, R. Cheng, and Z. Su, “Multi-label few-shot learning for aspect category detection,” in ACL/IJCNLP (1). Association for Computational Linguistics, 2021, pp. 6330–6340.
- [46] K. Cheng, S. Chou, and Y. Yang, “Multi-label few-shot learning for sound event recognition,” in MMSP. IEEE, 2019, pp. 1–5.
- [47] E. Law, K. West, M. I. Mandel, M. Bay, and J. S. Downie, “Evaluation of algorithms using games: The case of music tagging,” in ISMIR. International Society for Music Information Retrieval, 2009, pp. 387–392.
- [48] M. Defferrard, K. Benzi, P. Vandergheynst, and X. Bresson, “FMA: A dataset for music analysis,” in ISMIR, 2017, pp. 316–323.
- [49] C. Papaioannou, I. Valiantzas, T. Giannakopoulos, M. A. Kaliakatsos-Papakostas, and A. Potamianos, “A dataset for greek traditional and folk music: Lyra,” in ISMIR, 2022, pp. 377–383.
- [50] B. Uyar, H. S. Atli, S. Sentürk, B. Bozkurt, and X. Serra, “A corpus for computational research of turkish makam music,” in DLfM@JCDL. ACM, 2014, pp. 1–7.
- [51] S. Sentürk, “Computational analysis of audio recordings and music scores for the description and discovery of ottoman-turkish makam music,” Ph.D. dissertation, Pompeu Fabra University, Spain, 2017.
- [52] A. Srinivasamurthy, G. K. Koduri, S. Gulati, V. Ishwar, and X. Serra, “Corpora for music information research in indian art music,” in ICMC. Michigan Publishing, 2014.
- [53] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in ICLR, 2015.
- [54] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in ICLR, 2015.