# CultureMERT: Continual Pre-Training for Cross-Cultural Music Representation Learning
## Abstract
Recent advances in music foundation models have improved audio representation learning, yet their effectiveness across diverse musical traditions remains limited. We introduce CultureMERT-95M, a multi-culturally adapted foundation model developed to enhance cross-cultural music representation learning and understanding. To achieve this, we propose a two-stage continual pre-training strategy that integrates learning rate re-warming and re-decaying, enabling stable adaptation even with limited computational resources. Training on a 650-hour multi-cultural data mix, comprising Greek, Turkish, and Indian music traditions, results in an average improvement of 4.9% in ROC-AUC and AP across diverse non-Western music auto-tagging tasks, surpassing prior state-of-the-art, with minimal forgetting on Western-centric benchmarks. We further investigate task arithmetic, an alternative approach to multi-cultural adaptation that merges single-culture adapted models in the weight space. Task arithmetic performs on par with our multi-culturally trained model on non-Western auto-tagging tasks and shows no regression on Western datasets. Cross-cultural evaluation reveals that single-culture models transfer with varying effectiveness across musical traditions, whereas the multi-culturally adapted model achieves the best overall performance. To support research on world music representation learning, we publicly release CultureMERT-95M https://huggingface.co/ntua-slp/CultureMERT-95M and CultureMERT-TA-95M https://huggingface.co/ntua-slp/CultureMERT-TA-95M, fostering the development of more culturally aware music foundation models.
## 1 Introduction
Foundation models have recently emerged in the music domain [1, 2, 3, 4, 5], offering powerful general-purpose representations learned from large-scale audio data. These models capture broad musical characteristics and have demonstrated state-of-the-art performance across a range of music understanding tasks, reducing the need for task-specific training. By leveraging self-supervised learning (SSL) on large amounts of unlabelled music data, foundation models address data scarcity, reduce annotation costs, and improve generalization in music information retrieval (MIR) [4].
Despite these advances, most existing foundation models for music have been trained primarily on Western-centric datasets, limiting their ability to represent diverse musical styles [6, 7]. Many musical traditions, including Turkish, Indian, and Greek traditional music, feature unique melodic structures, modal or tonal systems, and rhythmic patterns that are not adequately captured by these models [8, 9, 10]. Failing to model such culture-specific stylistic elements not only narrows the applicability of music foundation models, for example, in region-specific recommendation systems [11] or cultural heritage preservation, but also overlooks rich, culturally specific knowledge crucial for advancing MIR research [4]. Accordingly, there is an urgent need to develop more inclusive and culturally aware computational models [12, 13], capable of generalizing beyond Western-centric traditions and adapting effectively to diverse underrepresented musical cultures.
One promising avenue for addressing these challenges is continual pre-training (CPT), which has emerged as an effective and increasingly popular approach in large language models (LLMs) [14, 15, 16, 17, 18, 19, 20, 21, 22] and multimodal learning [23]. By enabling models to incrementally adapt to new domains, tasks, or languages, CPT avoids the need for full re-training, which is often impractical and computationally expensive [18, 14, 19, 22, 24]. Notably, it has been shown to match, or even surpass, training from scratch in some cases [20, 21], while also converging faster [25] and mitigating catastrophic forgetting [26]. CPT has also gained traction in the audio domain, with recent work demonstrating its effectiveness in adapting pre-trained speech models to both high- and low-resource languages [24, 27, 28, 29, 30].
Additionally, model merging [31, 32, 33, 34] has proven to be a simple yet effective technique for adapting pre-trained models across multiple domains by combining domain-specific parameters in weight space, without requiring additional training [35] or access to the original training data [36]. A notable method within this paradigm is task arithmetic (TA) [37], which constructs task vectors by computing the difference between the parameters of an adapted model and its pre-trained counterpart, thereby encoding domain-specific knowledge. These task vectors can then be integrated into the pre-trained model via algebraic operations in Euclidean space to create a unified model from multiple independently adapted models.
While both continual pre-training and task arithmetic have been widely explored in other domains, their application to MIR remains largely unexplored. We bridge this gap by leveraging these techniques to adapt the MERT-v1-95M music foundation model [1], originally trained on 1K hours of predominantly Western music [1, 38], to diverse musical cultures from the Eastern Mediterranean and the Indian subcontinent, while preserving performance on "Western"-centric benchmarks.
We summarize our main contributions as follows:
1. To the best of our knowledge, this is the first study to explore continual pre-training and task arithmetic for cross-cultural adaptation in MIR, demonstrating their effectiveness in music audio representation learning.
1. We propose a two-stage CPT strategy that stabilizes training, mitigates catastrophic forgetting, and facilitates effective adaptation under constrained computational resources.
1. Our multi-cultural model, CultureMERT, outperforms the original MERT-v1 by an average of 4.9% across ROC-AUC and AP on culturally diverse non-Western music tagging tasks, while exhibiting minimal forgetting on Western benchmarks.
1. Our culturally adapted models surpass previous state-of-the-art results across all evaluated non-Western music tagging tasks.
1. We analyze cross-cultural transferability, showing that single-culture adaptations exhibit varying degrees of transfer across cultural domains.
To support reproducibility and further research in cross-cultural music representation learning, we publicly release CultureMERT-95M, along with the task arithmetic variant, CultureMERT-TA-95M.
## 2 Datasets
For our experiments, we use a diverse set of music datasets spanning both Western and non-Western traditions. Specifically, we adopt the MagnaTagATune (MTAT) [39] and FMA-medium [40] datasets to represent "Western" We use the term “Western” to refer to music styles predominantly rooted in Western cultures, including pop, rock, and Western classical. music. For "non-Western" traditions, we incorporate the Lyra corpus [41], featuring Greek traditional and folk music, along with three collections from the CompMusic Corpora https://compmusic.upf.edu/corpora [42]: Turkish-makam [43, 44], which, together with Lyra, represent music of the Eastern Mediterranean; and Hindustani and Carnatic music [45], representing North and South Indian classical traditions, respectively.
We assess our models on both Western and non-Western music tagging tasks for cross-cultural evaluation, using standard multi-label classification metrics, including the area under the receiver operating characteristic curve (ROC-AUC) and average precision (AP). Following [46, 47], we utilize the top-k tags relevant to each dataset: 50 tags for MTAT (spanning genre, instruments, and mood), 20 hierarchical genre tags for FMA-medium, 30 tags for Turkish-makam (covering makam, usul, and instruments), 20 tags for Hindustani and Carnatic (primarily reflecting raga, tala, instruments, and forms), and 30 tags for Lyra (related to genre, place, and instruments).
All audio is resampled to 24 kHz, and we adopt the same data splits as [46]. To prepare our data for continual pre-training, we extract 30-second segments from each training split of the non-Western datasets. Given the varying dataset sizes, we balance the pre-training duration across cultures to ensure proportional representation by extracting 200 hours each from the Turkish-makam, Carnatic, and Hindustani datasets, and 50 hours from Lyra due to its smaller size. Additionally, we combine these subsets to construct a unified 650-hour dataset integrating all four traditions for multi-cultural continual pre-training.
## 3 Method
The overall framework of our approach is illustrated in Figure 1, which depicts the two-stage continual pre-training strategy for CultureMERT. In this section, we first review the architecture and pre-training objective of MERT, and then present our CPT strategy for cultural adaptation. Finally, we investigate task arithmetic, an alternative approach to multi-cultural adaptation that merges culturally specialized models in weight space to construct a unified multi-cultural model, CultureMERT-TA.
### 3.1 MERT Pre-Training Objective
Our continual pre-training objective follows the self-supervised masked language modeling (MLM) objective of MERT RVQ-VAE, where two teacher models provide the pseudo-labels: (i) an acoustic teacher, the EnCodec model [48], which discretizes audio into tokens from $K=8$ residual vector quantization (RVQ) codebooks, each containing $C=1024$ codewords, and (ii) a musical teacher, based on constant-Q transform (CQT) spectrogram reconstruction, encoding pitch and harmonic structure.
MERT-v1-95M follows the HuBERT architecture [49], comprising a CNN-based feature extractor that encodes raw 24 kHz waveforms into 75 Hz frame-level representations, followed by a 12-layer Transformer encoder, producing 768-dimensional contextual embeddings. During training, a subset of frame embeddings is masked, and the model is optimized using a multi-task learning (MTL) objective, combining masked acoustic token prediction and spectrogram reconstruction. The overall training objective is:
$$
L=αL_RVQ+L_CQT, \tag{1}
$$
where the acoustic MLM loss $L_RVQ$ encourages the model to predict masked RVQ-VAE tokens from $K$ codebooks, using a noise-contrastive estimation (NCE) loss:
$$
L_RVQ=∑_k=1^K∑_t∈ M\log p_θ(c_t,k|
\boldsymbol{x}^\prime_t), \tag{2}
$$
with $M$ denoting the set of masked time frames, $c_t,k$ the ground-truth discrete codeword from the $k$ -th codebook at time frame $t$ extracted via the EnCodec tokenizer, and $p_θ$ the model’s predicted token distribution:
$$
p_θ(c|\boldsymbol{x}^\prime_t)=\frac{\exp(sim(T(\boldsymbol{
o}_t),\boldsymbol{e}_c)/τ)}{∑_c^\prime=1^C\exp(sim(T(
\boldsymbol{o}_t),\boldsymbol{e}_c^\prime)/τ)}. \tag{3}
$$
Here, $\boldsymbol{x}^\prime_t$ is the masked input feature, $\boldsymbol{o}_t$ is the model’s output representation, $T(\boldsymbol{o}_t)$ projects it to the codeword embedding space, $\boldsymbol{e}_c$ is the embedding of codeword $c∈C_k$ , where $k∈\{1,\dots,K\}$ , $sim(·,·)$ denotes cosine similarity, and $τ=0.1$ is a temperature scaling parameter.
The CQT reconstruction loss $L_CQT$ minimizes the mean squared error (MSE) between the model’s predicted $\hat{z}_CQT,t$ and ground-truth $z_CQT,t$ frame-level CQT features:
$$
L_CQT=∑_t∈ M≤ft\|z_CQT,t-\hat{
z}_CQT,t\right\|_2^2. \tag{4}
$$
### 3.2 Two-Stage Continual Pre-Training Strategy
To adapt the MERT foundation model to diverse musical traditions, we employ continual pre-training, which extends the training of a pre-trained model on new data, aiming to adapt it to a shifted domain or task while retaining prior knowledge, without re-training from scratch. In our case, this involves continually pre-training the MERT-v1-95M model, using the same pre-training objective, on culturally diverse data that introduce a significant distribution shift, as it was initially trained on predominantly Western music [1, 38]. Given this shift, naively continuing to train the model, i.e., adapting all parameters at once without resetting the learning rate, can lead to catastrophic forgetting [50] and poor adaptation [14], as confirmed by our preliminary experiments (see Table 1). To address this, we propose a two-stage strategy that stabilizes training through: (i) learning rate re-warming and re-decaying [14, 23, 21, 19, 51], and (ii) staged adaptation.
Staged Adaptation In our preliminary experiments, we observed an initial performance drop during CPT, followed by a slow recovery phase, a phenomenon known as the stability gap [52, 53, 23]. This instability arises due to the abrupt adaptation of model parameters to a substantially shifted data distribution, which can temporarily degrade previously learned representations before stabilizing. To mitigate this, rather than full-parameter adaptation on the entire dataset in a single epoch, which induces a large plasticity gradient for a long period [53], we split training into two stages to reduce instability and ensure smoother adaptation, as illustrated in Figure 1:
<details>
<summary>x1.png Details</summary>
### Visual Description
## Diagram: Multi-Stage MERT Architecture for Audio Processing
### Overview
The diagram illustrates a two-stage machine learning model (MERT) for processing multi-cultural audio data. It compares Stage 1 (100-hour data) and Stage 2 (650-hour data) architectures, highlighting differences in learning rate re-warming and component configurations. Key elements include transformer encoders, masked language modeling (MLM) losses, and 1D convolution feature extractors.
### Components/Axes
#### Stage 1 (100-hour Data)
- **Input**: 100-hour multi-cultural audio data (20% Music4All)
- **Components**:
- 1D Convolution Feature Extractor (orange)
- Transformer Encoder (blue dashed box)
- Music MLM Loss (green)
- Acoustic MLM Loss (blue)
- Codeword Embeddings (orange flame icon)
- **Learning Rate Graph**:
- X-axis: Epochs (0 to 100k)
- Y-axis: Learning Rate (5e-5 to 5e-4)
- Legend: Green = 100-hour data (20% Music4All)
- Warm-up: 10% (steep initial rise)
#### Stage 2 (650-hour Data)
- **Input**: 650-hour multi-cultural audio data
- **Components**:
- 1D Convolution Feature Extractor (orange)
- Transformer Encoder (red solid box)
- Music MLM Loss (green)
- Acoustic MLM Loss (blue)
- Codeword Embeddings (orange flame icon)
- **Learning Rate Graph**:
- X-axis: Epochs (0 to 100k)
- Y-axis: Learning Rate (5e-6 to 5e-5)
- Legend: Red = 650-hour data
- Warm-up: 1% (gentle initial rise)
### Detailed Analysis
1. **Learning Rate Re-Warming**:
- Stage 1 shows a rapid warm-up (10%) with a peak at ~5e-4, followed by decay.
- Stage 2 uses a slower 1% warm-up, peaking at ~5e-5, indicating scaled training for larger datasets.
2. **Component Flow**:
- Both stages share identical component order: 1D convolution → Transformer Encoder → MLM losses → Codeword Embeddings.
- Stage 2's transformer encoder is visually emphasized with a red box and flame icon, suggesting enhanced optimization.
3. **Data Representation**:
- Audio waveforms (colored lines) feed into the 1D convolution layer.
- Loss functions (green/blue) are positioned above the transformer encoder, indicating backpropagation paths.
### Key Observations
- **Warm-up Duration**: Stage 2's 1% warm-up contrasts with Stage 1's 10%, reflecting adjustments for dataset size.
- **Component Consistency**: Identical architecture across stages suggests incremental scaling rather than structural changes.
- **Loss Function Balance**: Equal placement of Music and Acoustic MLM losses implies multi-task learning for domain generalization.
### Interpretation
The diagram demonstrates a progressive training strategy where Stage 2 optimizes for larger, more diverse datasets through:
1. **Extended Warm-up**: Gradual learning rate increases prevent instability with 650-hour data.
2. **Component Emphasis**: The red-highlighted transformer encoder in Stage 2 may indicate deeper fine-tuning or regularization.
3. **Multi-Cultural Focus**: The explicit separation of Music4All data (20% in Stage 1) suggests targeted improvements for musical audio processing.
The architecture prioritizes stability (via controlled warm-up) and scalability (identical components across stages), enabling efficient training on both small and large-scale audio corpora. The flame icons on codeword embeddings likely represent critical components for semantic representation in audio-text tasks.
</details>
Figure 1: Two-Stage Continual Pre-Training Strategy for CultureMERT. In Stage 1, a subset of parameters is trained on 100h of multi-cultural data with 20% Western music for stabilization. In Stage 2, all parameters are unfrozen and trained on the full 650h dataset. Learning rate re-warming and re-decaying is applied in both stages.
Stage 1 Stabilization Phase: We first train on a smaller data subset [52], updating only the CNN-based feature extractor and the codeword embedding layer while keeping the Transformer encoder frozen. To reduce the distribution gap and mitigate forgetting [19, 28, 25], we incorporate a fraction of Music4All data [54], which is primarily of Western origin, into the pre-training mix, accounting for 20% of the total training data (Western replay).
Stage 2 Full Adaptation: We unfreeze the Transformer encoder and continue training on the full dataset.
| CPT Strategy | Western Replay | Turkish-makam | MTAT |
| --- | --- | --- | --- |
| MERT-v1 (Baseline) | - | 83.2 | 89.6 |
| Single-stage | ✓ | 83.8 | 86.0 |
| Single-stage (no re-warm) | ✓ | 83.0 | 87.5 |
| Two-stage (Ours) | Stage 1 | 89.6 | 89.2 |
| Two-stage (Ours) | Both stages | 88.6 | 89.4 |
Table 1: CPT Strategy Comparison. ROC-AUC scores on Turkish-makam and MTAT datasets. Two-stage CPT outperforms single-stage adaptation, with Western replay limited to Stage 1 yielding the best trade-off between cultural adaptation and knowledge retention.
This two-stage approach is particularly motivated by computational constraints, specifically the batch size mismatch between pre-training and adaptation. MERT-v1-95M was originally trained with batch sizes of 1.5 hours per step, whereas we use a significantly smaller effective batch size of 160 seconds per step due to memory limitations. Training with this reduced batch size directly on the entire dataset with full-parameter adaptation resulted in unstable training and frequent crashes, degrading performance on both Western and non-Western benchmarks. By structuring adaptation in two stages, we strike to balance plasticity (adaptation to non-Western traditions) and stability (retaining knowledge on Western datasets), a challenge known as the stability-plasticity dilemma [55, 56]. Intuitively, the initial stabilization phase allows lower-level acoustic representations, captured by the CNN-based feature extractor and the codeword embeddings, to adapt first and calibrate to the shifted distribution before updating high-level Transformer representations.
Learning Rate Re-Warming To further improve adaptation stability, we apply learning rate re-warming and re-decaying in both stages. Prior work has shown that resetting the learning rate schedule, i.e., re-warming the model, during continual pre-training is crucial for preventing poor convergence and mitigating catastrophic forgetting [14, 23, 21, 19, 51]. In Stage 1, we adopt a moderately aggressive warm-up and decay schedule to encourage early adaptation of low-level representations. In Stage 2, a less aggressive schedule balances plasticity and stability during full-model training, reducing also training instabilities.
Following this two-stage CPT strategy, we develop two types of culturally adapted models: (i) a multi-culturally adapted model, CultureMERT, trained on a culturally diverse mix spanning all four non-Western musical traditions; and (ii) single-culture adapted models, each continually pre-trained on data from a single tradition, resulting in MakamMERT, HindustaniMERT, CarnaticMERT, and LyraMERT.
### 3.3 Task Arithmetic for Cross-Cultural Adaptation
As an alternative to continual pre-training on multi-cultural data, we explore task arithmetic [37], a model merging method that combines culturally specialized models in weight space to construct a unified multi-cultural model. Task arithmetic operates by algebraically merging model parameters through task vector addition and negation.
In our setting, we obtain task vectors by computing the element-wise difference between the parameters of the single-culture continually pre-trained models and those of the MERT-v1 model. Formally, given the pre-trained base model with parameters $θ_pre$ and a continually pre-trained model $θ_i$ adapted to a cultural dataset $D_i$ , the task vector for culture $i$ is given by $τ_i=θ_i-θ_pre$ , capturing the parameter shift induced by culture-specific adaptation.
For multi-cultural adaptation, we construct a unified model $θ_merged$ by merging $N$ single-culture adapted models via task arithmetic, summing their respective task vectors $τ_i$ with corresponding scaling factors $λ_i$ :
$$
θ_merged=θ_pre+∑_i=1^Nλ_iτ_i, \tag{5}
$$
where $λ_i∈ℝ$ are scalar hyperparameters that control the contribution of each task vector. Prior work typically uses a single scaling factor $λ$ for all task vectors, i.e., $λ_i=λ$ , $∀ i$ . In the special case where $λ=1/N$ , Equation 5 simplifies to weight averaging [33, 34, 31], in which the adapted models are merged by directly averaging their parameters.
## 4 Experiments
### 4.1 Implementation Details
In all continual pre-training setups, we initialize our models from the publicly available MERT-v1-95M https://huggingface.co/m-a-p/MERT-v1-95M pre-trained checkpoint. Training was conducted using the fairseq https://github.com/facebookresearch/fairseq framework on a single NVIDIA GeForce GTX TITAN X GPU with 12 GB of memory. All models were trained with half-precision (FP16), using 5-second audio segments as input context, randomly cropped from the extracted 30-second pre-training audio data. The weight of the acoustic loss in the pre-training objective is set to $α=10.0$ . The EnCodec neural audio codec (NAC) model [48], which tokenizes audio into discrete codewords, remains frozen during continual pre-training, as in [1]. To enhance representation robustness, we apply in-batch noise mixture augmentation with a mixup probability of $0.5$ , and use pre-layer normalization (Pre-LN) [57] for training stability, following [1]. Other training settings mirror those of the MERT-v1-95M setup.
### 4.2 Probing-Based Evaluation
Following [58, 1, 2], we adopt a probing-based evaluation rather than fine-tuning, keeping the pre-trained models frozen as deep feature extractors while training only a shallow multilayer perceptron (MLP) with a single 512-dimensional hidden layer for sequence-level tasks. Our evaluation follows the MARBLE protocol [59] under constrained settings, and we apply it to both Western and non-Western music tagging tasks for cross-cultural evaluation. To process long-duration audio files, we segment them into 30-second chunks using a sliding window approach and aggregate the chunk-level predictions by averaging to obtain the final prediction for the entire audio file. For Turkish-makam, Hindustani, and Carnatic tasks, we apply a maximum duration cut as in [46] to ensure comparability with prior state-of-the-art results.
### 4.3 Continual Pre-Training Settings
| Dataset | Turkish-makam | Hindustani | Carnatic | Lyra | FMA-medium | MagnaTagATune | Avg. | | | | | | |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Metrics | ROC | AP | ROC | AP | ROC | AP | ROC | AP | ROC | AP | ROC | AP | |
| MERT-v1 | 83.2 0.08 | 53.3 0.12 | 82.4 0.04 | 52.9 0.19 | 74.9 0.05 | 39.7 0.15 | 85.7 0.10 | 56.5 0.18 | 90.7 0.04 | 48.1 0.11 | 89.6 0.07 | 35.9 0.15 | 66.1 |
| MakamMERT | 88.7 0.11 | 58.8 0.22 | 84.5 0.16 | 57.8 0.18 | 77.6 0.14 | 42.7 0.16 | 84.6 0.12 | 53.2 0.17 | 90.3 0.12 | 47.1 0.16 | 89.0 0.07 | 35.6 0.12 | 67.5 |
| CarnaticMERT | 88.4 0.06 | 58.4 0.16 | 87.0 0.06 | 60.2 0.14 | 78.8 0.13 | 44.0 0.17 | 85.4 0.11 | 55.8 0.16 | 90.2 0.10 | 46.7 0.09 | 89.2 0.10 | 35.3 0.11 | 68.3 |
| HindustaniMERT | 88.3 0.12 | 58.2 0.16 | 87.4 0.11 | 60.3 0.16 | 77.0 0.12 | 42.7 0.16 | 84.2 0.13 | 52.0 0.15 | 90.2 0.13 | 46.1 0.10 | 89.1 0.09 | 35.8 0.13 | 67.6 |
| LyraMERT | 86.7 0.07 | 56.8 0.13 | 85.9 0.08 | 57.4 0.13 | 76.4 0.09 | 40.1 0.13 | 85.0 0.11 | 53.5 0.14 | 90.0 0.08 | 46.0 0.16 | 88.9 0.05 | 35.1 0.14 | 66.8 |
| CultureMERT | 89.6 0.09 | 60.6 0.21 | 88.2 0.20 | 63.5 0.24 | 79.2 0.18 | 43.1 0.22 | 86.9 0.10 | 56.7 0.20 | 90.7 0.09 | 48.1 0.13 | 89.4 0.09 | 35.9 0.16 | 69.3 |
| CultureMERT-TA | 89.0 0.12 | 61.0 0.18 | 87.5 0.10 | 59.3 0.13 | 79.1 0.11 | 43.3 0.13 | 87.3 0.08 | 57.3 0.19 | 90.8 0.06 | 49.1 0.15 | 89.6 0.10 | 36.4 0.14 | 69.1 |
| (Previous) SOTA | 87.7 [46] | 57.7 [46] | 86.5 [46] | 63.1 [46] | 77.0 [46] | 43.9 [46] | 85.4 [46] | 54.3 [46] | 92.4 [46] | 53.7 [46] | 92.7 [60] | 41.4 [58] | - |
Table 2: Evaluation Results (ROC-AUC and AP) of Pre-Trained and Culturally Adapted MERT Models on Diverse Music Auto-Tagging Tasks. We report averages across five random seeds with standard deviations as subscripts. The "Avg." column represents the average performance across all datasets and evaluation metrics for each model. The results highlight the impact of multi-cultural CPT and model merging via task arithmetic on cross-cultural adaptation and transfer.
Multi-Cultural CPT In Stage 1, training runs for 2,250 steps with a 10% linear warm-up period, using 100 hours of the dataset. Optimization follows AdamW [61] with $β_1=0.9$ , $β_2=0.999$ , and $ε=1e^-5$ . Training employs an effective batch size of 32 recordings (160 seconds), with gradient accumulation over 8 steps. The maximum learning rate is set to $η_max=5e^-4$ , followed by a cosine decay to a minimum of $η_min=5e^-5$ . Gradient clipping is applied with a norm of $1.0$ to prevent exploding gradients. In Stage 2, training extends to 14,625 steps with a 1% warm-up period, using the full 650-hour dataset. Optimization follows AdamW with $β_1=0.9$ , $β_2=0.95$ , and $ε=1e^-5$ , maintaining the same batch size as Stage 1. The learning rate decays from a maximum value of $η_max=5e^-5$ to $η_min=5e^-6$ . Gradient clipping remains at $1.0$ .
Single-Culture CPT In Stage 1, we train on 60 hours for a total of 1,350 training steps. In Stage 2, we expand training to the full 200-hour dataset for 4,500 steps. We employ the same optimizers, batch size, and learning rate schedules as in the multi-cultural CPT. For Lyra, due to its smaller size (50 hours), we train on 20 hours in Stage 1 (450 steps) and then on the full dataset in Stage 2 (1,125 steps).
<details>
<summary>x2.png Details</summary>
### Visual Description
## Radar Chart: Multilingual MERT Performance Comparison
### Overview
The image is a radar chart comparing the performance of seven multilingual MERT (Multilingual Encoder Representations from Transformers) models across six linguistic/cultural domains. The chart uses overlapping polygons to visualize relative performance metrics, with axes labeled as Carnatic, Hindustani, Turkish-makam, Lyra, FMA-medium, and MagnaTagATune. The legend identifies models by color-coded lines.
### Components/Axes
- **Axes (Features)**:
1. Carnatic (top-left)
2. Hindustani (top-right)
3. Turkish-makam (right)
4. Lyra (bottom-left)
5. FMA-medium (bottom-center)
6. MagnaTagATune (bottom-right)
- **Legend (Models)**:
- MERT-v1 (blue)
- CultureMERT (orange)
- MakamMERT (teal)
- CarnaticMERT (red)
- HindustaniMERT (purple)
- LyraMERT (yellow)
- CultureMERT-TA (green)
- **Placement**:
- Legend: Right side of the chart
- Axes: Arranged radially around the center
- Data lines: Polygons connecting axis points
### Detailed Analysis
- **Model Performance**:
- **CultureMERT-TA (green)**: Dominates in Turkish-makam (~0.9) and FMA-medium (~0.8), with moderate performance in other domains.
- **HindustaniMERT (purple)**: Peaks in Hindustani (~0.85) and Carnatic (~0.7), with weaker performance in Turkish-makam (~0.4).
- **LyraMERT (yellow)**: Strong in Lyra (~0.75) and FMA-medium (~0.6), but low in Turkish-makam (~0.3).
- **MERT-v1 (blue)**: Lowest overall scores (~0.3–0.5), with a notable peak in MagnaTagATune (~0.6).
- **CarnaticMERT (red)**: Balanced performance in Carnatic (~0.7) and Hindustani (~0.6), weaker in Turkish-makam (~0.4).
- **MakamMERT (teal)**: Moderate across all domains, with a peak in Turkish-makam (~0.65).
- **CultureMERT (orange)**: Broad coverage, strongest in Carnatic (~0.8) and Hindustani (~0.7), weaker in Lyra (~0.4).
### Key Observations
1. **Specialization vs. Generalization**:
- CultureMERT-TA and HindustaniMERT excel in culturally specific domains (Turkish-makam, Hindustani), suggesting domain-specific tuning improves performance.
- MERT-v1 underperforms but shows unexpected strength in MagnaTagATune (~0.6), possibly due to generalist training.
2. **Overlap and Trade-offs**:
- LyraMERT and CultureMERT-TA overlap significantly in FMA-medium (~0.6–0.8), indicating similar capabilities in this domain.
- Turkish-makam is a weak point for most models except CultureMERT-TA and MakamMERT.
3. **Anomalies**:
- MERT-v1’s peak in MagnaTagATune (~0.6) contrasts with its low scores elsewhere, suggesting potential overfitting or domain-specific biases.
### Interpretation
The chart highlights the importance of model specialization for cultural and linguistic domains. CultureMERT-TA and HindustaniMERT demonstrate superior performance in their respective domains, while MERT-v1’s generalist approach yields mediocre results. The unexpected strength of MERT-v1 in MagnaTagATune warrants further investigation into its training data or architecture. For applications requiring cross-cultural adaptability, hybrid models combining domain-specific and generalist components may be optimal. The chart underscores the need for targeted evaluation metrics when deploying multilingual models in culturally specific contexts.
</details>
Figure 2: Cross-Cultural Transferability. Relative ROC-AUC performance across datasets, highlighting key trends in cross-cultural transfer. CultureMERT generalizes well to non-Western datasets, while task arithmetic performs on par in these settings and even surpasses both the pre-trained and multi-culturally adapted models on Western benchmarks (FMA-medium, MTAT) and Lyra.
## 5 Results and Discussion
As shown in Table 2, CultureMERT, adapted via multi-cultural continual pre-training, consistently outperforms the original MERT-v1 model across all non-Western tasks and evaluation metrics, achieving an average improvement of 4.9%. It also surpasses the single-culture adapted models on average, suggesting that incorporating culturally diverse data during CPT benefits all non-Western traditions by improving the quality of representations for each individual culture, thereby enhancing generalization. Notably, CultureMERT achieves this with minimal forgetting on Western benchmarks (0.05% average drop across ROC-AUC and AP), demonstrating the efficacy of our approach. We further observe that single-culture adapted models tend to perform best on their respective in-domain tasks for well-resourced traditions, reaffirming the effectiveness of CPT for domain-specific adaptation [18]. However, even low-resource adaptation, as in the case of LyraMERT trained on just 50 hours, leads to noticeable gains across other non-Western tasks, indicating that even limited cultural exposure can significantly boost cross-cultural generalization. Moreover, task arithmetic performs comparably to CultureMERT on non-Western tasks and even surpasses it on Western benchmarks and Lyra, demonstrating that weight-space merging of culturally specialized models can serve as an effective, training-free alternative to multi-cultural CPT—provided such models are available. Interestingly, it also outperforms the unadapted base model by 0.4% on average across Western tasks. Notably, only the multi-cultural models, CultureMERT and CultureMERT-TA, outperform MERT-v1 on Lyra, where the latter already serves as a strong baseline. This further underscores the effectiveness of multi-cultural adaptation, particularly in low-resource and transfer settings. Finally, CultureMERT and CultureMERT-TA surpass previous state-of-the-art (SOTA) results on all non-Western music tagging tasks, with the best task arithmetic variant obtained using $λ=0.2$ (see Figure 4).
### 5.1 Cross-Cultural Transfer
As illustrated in Figure 2, continual pre-training on one musical tradition can benefit others to varying degrees, revealing asymmetries in cross-cultural transfer effectiveness. For instance, we observe strong transfer between Turkish-makam and Carnatic music, with models adapted to either tradition generalizing well to the other. This aligns with their shared theoretical foundations as modal frameworks that emphasize microtonality and improvisation, serving similar roles in their respective cultures [62]. Additionally, the strong performance of the Carnatic-adapted model on the Hindustani domain reinforces the musical proximity between these traditions, particularly in their shared use of raga (melodic mode) and tala (rhythmic framework) [10]. Interestingly, the model adapted to Carnatic music appears to be the most consistently transferable among single-culture adaptations, achieving strong results not only within Indian classical traditions but also generalizing well to Turkish-makam and Lyra.
### 5.2 Token-Level Culture Similarity
To further examine cross-cultural similarities in our data, we analyze token overlap across musical traditions using both the Jensen-Shannon divergence (JSD) and cosine distance between token distributions extracted from the EnCodec model [48], which serves as our audio tokenizer. Lower values in both metrics indicate greater similarity. Our analysis, as shown in Figure 3, reveals strong token-level similarity among non-Western traditions, particularly between Hindustani and Carnatic music. In contrast, Western datasets (MTAT, FMA-medium) are highly similar to each other but notably dissimilar from non-Western traditions. Greek traditional music (Lyra), while distinct, aligns more closely with non-Western traditions than Western ones. Interestingly, these findings correlate with our results on cross-cultural transfer (Section 5.1), suggesting that token-level similarity metrics can serve as predictors of positive cross-cultural transfer. This insight has practical implications: such similarity metrics can guide the selection and refinement of pre-training data mixtures during CPT, or inform the adjustment of arithmetic operations when merging models via task arithmetic. Similar approaches for quantifying language similarity and predicting positive cross-lingual transfer, based on the similarity of extracted linguistic or acoustic tokens, have been explored in both the text [17, 63] and speech domains [29].
<details>
<summary>x3.png Details</summary>
### Visual Description
## Heatmap: Comparison of Text Similarity Metrics
### Overview
The image is a heatmap comparing six text similarity metrics (Turkish-makam, Hindustani, Carnatic, Lyra, MTAT, FMA-medium) using two distance measures: Jensen-Shannon Divergence (blue) and Cosine Distance (green). Values represent percentage differences between metric pairs, with darker colors indicating higher divergence/distance.
### Components/Axes
- **X/Y Axes**: Labeled with the six metrics (Turkish-makam, Hindustani, Carnatic, Lyra, MTAT, FMA-medium).
- **Legend**:
- Blue squares = Jensen-Shannon Divergence
- Green squares = Cosine Distance
- **Values**: Percentages (0–22.6%) embedded in cells, with empty diagonal cells (self-comparisons).
### Detailed Analysis
#### Jensen-Shannon Divergence (Blue)
- **Highest Value**: MTAT vs MTAT (22.6%)
- **Lowest Value**: Hindustani vs Hindustani (3.6%)
- **Trend**: Diagonal cells (self-comparisons) are empty. Off-diagonal values show MTAT consistently has the highest divergence (e.g., MTAT vs Hindustani: 19.8%, MTAT vs Carnatic: 22.5%).
#### Cosine Distance (Green)
- **Highest Value**: Turkish-makam vs MTAT (19.8%)
- **Lowest Value**: MTAT vs MTAT (2.1%)
- **Trend**: Diagonal cells are empty. Turkish-makam shows the highest distance to MTAT, while MTAT has the smallest self-distance.
### Key Observations
1. **MTAT Dominance**: MTAT has the highest Jensen-Shannon Divergence (22.6%) and lowest Cosine Distance (2.1%) when compared to itself, suggesting it is the most variable metric.
2. **Turkish-makam vs MTAT**: The largest Cosine Distance (19.8%) indicates these metrics are the most dissimilar.
3. **Diagonal Gaps**: Empty diagonal cells imply self-comparisons are excluded or normalized to 0.
4. **Color Consistency**: Blue (Jensen-Shannon) and green (Cosine) values align with legend expectations (e.g., MTAT’s high blue values match its high divergence).
### Interpretation
The heatmap reveals that **MTAT** is the most variable metric (highest Jensen-Shannon Divergence) and most consistent within itself (lowest Cosine Distance). **Turkish-makam** and **MTAT** are the most dissimilar (highest Cosine Distance). The absence of diagonal values suggests self-comparisons are excluded, focusing on inter-metric differences. This could reflect how these metrics capture distinct linguistic or structural features in text analysis.
</details>
Figure 3: Token Similarity Across Cultures. Pairwise similarity between acoustic token distributions extracted from the EnCodec NAC model [48]. Similarity scores are averaged across $8$ codebooks, each containing $1024$ discrete codewords (acoustic pseudo-tokens).
### 5.3 Task Arithmetic Scaling Factor
A key consideration in task arithmetic is the choice of the scaling factor $λ$ , which controls the balance between task vectors. Prior work [64, 65] has shown that suboptimal values can significantly degrade performance in multi-task model merging. We systematically evaluate different values of a shared scaling factor $λ∈\{0.1,0.2,0.25,0.3,0.5,0.75,1.0\}$ , applied uniformly across all task vectors, including the special case of weight averaging ( $λ=0.25$ ). We similarly observe a consistent trend: ill-suited values, such as $λ=1.0$ , result in poor performance across all benchmarks, as shown in Figure 4.
<details>
<summary>x4.png Details</summary>
### Visual Description
## Line Graph: ROC-AUC Performance vs. Scaling Factor λ
### Overview
The image is a line graph comparing the ROC-AUC (%) performance of six audio processing methods across different scaling factors (λ). The x-axis represents the scaling factor (λ) ranging from 0.1 to 1.0, while the y-axis shows ROC-AUC performance (70–90%). Six data series are plotted with distinct line styles and markers, each corresponding to a specific method.
### Components/Axes
- **X-axis**: Scaling Factor λ (0.1, 0.2, 0.25, 0.3, 0.5, 0.75, 1.0)
- **Y-axis**: ROC-AUC (%) (70–90% in 5% increments)
- **Legend**: Located at the bottom-left, mapping colors/markers to methods:
- **Turkish-makam**: Green circles (dashed line)
- **Lyra**: Orange triangles (dotted line)
- **Hindustani**: Pink squares (dash-dot line)
- **MagnaTagATune**: Blue plus signs (solid line)
- **FMA-medium**: Light blue crosses (dashed line)
- **Carnatic**: Orange diamonds (dotted line)
### Detailed Analysis
1. **Turkish-makam** (Green circles):
- Starts at ~88% ROC-AUC at λ=0.1, declines steadily to ~84% at λ=1.0.
- Trend: Gradual downward slope.
2. **Lyra** (Orange triangles):
- Peaks at ~87% at λ=0.2, then drops sharply to ~73% at λ=1.0.
- Trend: Initial rise followed by steep decline.
3. **Hindustani** (Pink squares):
- Starts at ~86% at λ=0.1, dips to ~82% at λ=0.5, then stabilizes at ~78% at λ=1.0.
- Trend: Moderate decline with plateau.
4. **MagnaTagATune** (Blue plus signs):
- Maintains ~89% ROC-AUC across all λ values, with minimal fluctuation.
- Trend: Flat line.
5. **FMA-medium** (Light blue crosses):
- Consistently highest performance (~90% at λ=0.1, ~86% at λ=1.0).
- Trend: Slight downward slope.
6. **Carnatic** (Orange diamonds):
- Declines from ~78% at λ=0.1 to ~71% at λ=1.0.
- Trend: Steady downward slope.
### Key Observations
- **FMA-medium** and **MagnaTagATune** exhibit the most stable performance, with FMA-medium achieving the highest ROC-AUC values.
- **Lyra** and **Carnatic** show significant degradation as λ increases, with Lyra dropping ~14% and Carnatic dropping ~7%.
- **Turkish-makam** and **Hindustani** demonstrate moderate declines, with Turkish-makam retaining higher performance than Hindustani at higher λ values.
- **Lyra** and **Hindustani** intersect at λ=0.25 (~86% ROC-AUC), indicating crossover in performance.
### Interpretation
The data suggests that audio processing methods vary in robustness to scaling factor adjustments. **FMA-medium** and **MagnaTagATune** are the most resilient, maintaining high ROC-AUC across all λ values. In contrast, **Lyra** and **Carnatic** are highly sensitive to scaling, with performance dropping sharply as λ increases. The crossover between Lyra and Hindustani at λ=0.25 highlights that optimal scaling factors are method-dependent. These trends imply that method selection should consider the expected λ range in deployment scenarios. The consistent performance of FMA-medium and MagnaTagATune positions them as preferred choices for applications requiring stability under scaling variations.
</details>
Figure 4: Effect of Scaling Factor $λ$ on Task Arithmetic Performance. The ROC-AUC scores across six diverse music tagging tasks demonstrate how varying $λ$ impacts task arithmetic when merging the four non-Western single-culture adapted models.
## 6 Conclusions
In this paper, we introduce CultureMERT-95M, a multi-culturally adapted music foundation model developed via continual pre-training on diverse non-Western musical traditions. We propose a two-stage CPT strategy that incorporates learning rate re-warming and staged adaptation for stable training. Cross-cultural evaluation demonstrates that CultureMERT-95M consistently outperforms the base MERT-v1-95M model on non-Western music tagging tasks, surpassing prior state-of-the-art methods while preserving performance on Western benchmarks. Additionally, we investigate task arithmetic, which offers a strong alternative to multi-cultural CPT by effectively merging culturally specialized models in weight space.
While our results are promising, several limitations remain. The frozen EnCodec tokenizer used in the MERT architecture may be suboptimal for encoding culturally diverse musical languages, as it was pre-trained on Western music. Future directions include scaling to additional musical cultures, exploring alternative architectures, extending evaluation beyond sequence-level classification tasks, conducting fine-grained ablation studies, and investigating whether the proposed two-stage CPT strategy remains necessary under less constrained computational budgets.
## 7 Ethics Statement
### 7.1 Cultural Framing and Interpretive Scope
We acknowledge the limitations of framing music within a "Western" versus "non-Western" dichotomy. While such terminology is commonly used in computational research for convenience, it risks oversimplifying the rich diversity of global musical traditions. Furthermore, this work does not aim to establish or analyze cross-cultural similarities from an ethnomusicological perspective. Our analysis of cross-cultural transferability should be considered in light of potential limitations in the representativeness and coverage of the corpora used.
### 7.2 Responsible Use
Careful consideration is advised before deploying these models in real-world contexts, as they may still reflect cultural and dataset-specific biases. Some of the datasets used in this work are not publicly available and were obtained under research-use agreements. The released models should not be used in commercial or generative applications without explicit attention to cultural representation, appropriate licensing, and the consent of the relevant communities or dataset curators.
## 8 Acknowledgments
We would like to thank the reviewers and Georgios Paraskevopoulos for their valuable and constructive feedback, which helped us improve this work. We also gratefully acknowledge the Music Technology Group (MTG) at Universitat Pompeu Fabra for providing access to datasets used in this study. This work has been partially supported by project MIS 5154714 of the National Recovery and Resilience Plan Greece 2.0 funded by the European Union under the NextGenerationEU Program.
## References
- [1] Y. Li, R. Yuan, G. Zhang, Y. Ma, X. Chen, H. Yin, C. Xiao, C. Lin, A. Ragni, E. Benetos et al., “MERT: acoustic music understanding model with large-scale self-supervised training,” in The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
- [2] M. Won, Y. Hung, and D. Le, “A foundation model for music informatics,” in IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2024, Seoul, Republic of Korea, April 14-19, 2024. IEEE, 2024, pp. 1226–1230.
- [3] P. Dhariwal, H. Jun, C. Payne, J. W. Kim, A. Radford, and I. Sutskever, “Jukebox: A generative model for music,” CoRR, vol. abs/2005.00341, 2020.
- [4] Y. Ma, A. Øland, A. Ragni, B. M. D. Sette, C. Saitis, C. Donahue, C. Lin, C. Plachouras, E. Benetos, E. Quinton et al., “Foundation models for music: A survey,” CoRR, vol. abs/2408.14340, 2024.
- [5] W. Li, Y. Cai, Z. Wu, W. Zhang, Y. Chen, R. Qi, M. Dong, P. Chen, X. Dong, F. Shi et al., “A survey of foundation models for music understanding,” CoRR, vol. abs/2409.09601, 2024.
- [6] E. Gómez, P. Herrera, and F. Gómez-Martin, “Computational Ethnomusicology: perspectives and challenges,” Journal of New Music Research, vol. 42, no. 2, June 2013, pp. 111–112.
- [7] A. Mehta, S. Chauhan, A. Djanibekov, A. Kulkarni, G. Xia, and M. Choudhury, “Music for all: Exploring multicultural representations in music generation models,” CoRR, vol. abs/2502.07328, 2025.
- [8] T. Lidy, C. N. S. Jr., O. Cornelis, F. Gouyon, A. Rauber, C. A. A. Kaestner, and A. L. Koerich, “On the suitability of state-of-the-art music information retrieval methods for analyzing, categorizing and accessing non-western and ethnic music collections,” Signal Process., vol. 90, no. 4, 2010, pp. 1032–1048.
- [9] G. Plaja-Roglans, T. Nuttall, L. Pearson, X. Serra, and M. Miron, “Repertoire-specific vocal pitch data generation for improved melodic analysis of carnatic music,” Trans. Int. Soc. Music. Inf. Retr., vol. 6, no. 1, 2023, pp. 13–26.
- [10] G. K. Koduri, M. Miron, J. Serrà, and X. Serra, “Computational approaches for the understanding of melody in carnatic music,” in Proceedings of the 12th International Society for Music Information Retrieval Conference, ISMIR 2011, Miami, Florida, USA, October 24-28, 2011, A. Klapuri and C. Leider, Eds. University of Miami, 2011, pp. 263–268.
- [11] A. Ferraro, G. Ferreira, F. Diaz, and G. Born, “Measuring commonality in recommendation of cultural content to strengthen cultural citizenship,” Trans. Recomm. Syst., vol. 2, no. 1, 2024, pp. 10:1–10:32.
- [12] C. C. Liu, I. Gurevych, and A. Korhonen, “Culturally aware and adapted NLP: A taxonomy and a survey of the state of the art,” CoRR, vol. abs/2406.03930, 2024.
- [13] A. Holzapfel, B. L. Sturm, and M. Coeckelbergh, “Ethical dimensions of music information retrieval technology,” Trans. Int. Soc. Music. Inf. Retr., vol. 1, no. 1, 2018, pp. 44–55.
- [14] A. Ibrahim, B. Thérien, K. Gupta, M. L. Richter, Q. G. Anthony, E. Belilovsky, T. Lesort, and I. Rish, “Simple and scalable strategies to continually pre-train large language models,” Trans. Mach. Learn. Res., vol. 2024, 2024.
- [15] D. M. Alves, J. Pombal, N. M. Guerreiro, P. H. Martins, J. Alves, M. A. Farajian, B. Peters, R. Rei, P. Fernandes, S. Agrawal et al., “Tower: An open multilingual large language model for translation-related tasks,” CoRR, vol. abs/2402.17733, 2024.
- [16] L. Voukoutis, D. Roussis, G. Paraskevopoulos, S. Sofianopoulos, P. Prokopidis, V. Papavasileiou, A. Katsamanis, S. Piperidis, and V. Katsouros, “Meltemi: The first open large language model for greek,” CoRR, vol. abs/2407.20743, 2024.
- [17] E. Gogoulou, T. Lesort, M. Boman, and J. Nivre, “Continual learning under language shift,” in Text, Speech, and Dialogue - 27th International Conference, TSD 2024, Brno, Czech Republic, September 9-13, 2024, Proceedings, Part I, ser. Lecture Notes in Computer Science, E. Nöth, A. Horák, and P. Sojka, Eds., vol. 15048. Springer, 2024, pp. 71–84.
- [18] S. Gururangan, A. Marasovic, S. Swayamdipta, K. Lo, I. Beltagy, D. Downey, and N. A. Smith, “Don’t stop pretraining: Adapt language models to domains and tasks,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, D. Jurafsky, J. Chai, N. Schluter, and J. R. Tetreault, Eds. Association for Computational Linguistics, 2020, pp. 8342–8360.
- [19] J. Parmar, S. Satheesh, M. Patwary, M. Shoeybi, and B. Catanzaro, “Reuse, don’t retrain: A recipe for continued pretraining of language models,” CoRR, vol. abs/2407.07263, 2024.
- [20] K. Fujii, T. Nakamura, M. Loem, H. Iida, M. Ohi, K. Hattori, H. Shota, S. Mizuki, R. Yokota, and N. Okazaki, “Continual pre-training for cross-lingual LLM adaptation: Enhancing japanese language capabilities,” CoRR, vol. abs/2404.17790, 2024.
- [21] K. Gupta, B. Thérien, A. Ibrahim, M. L. Richter, Q. Anthony, E. Belilovsky, I. Rish, and T. Lesort, “Continual pre-training of large language models: How to (re)warm your model?” CoRR, vol. abs/2308.04014, 2023.
- [22] H. Shi, Z. Xu, H. Wang, W. Qin, W. Wang, Y. Wang, and H. Wang, “Continual learning of large language models: A comprehensive survey,” CoRR, vol. abs/2404.16789, 2024.
- [23] V. Udandarao, K. Roth, S. Dziadzio, A. Prabhu, M. Cherti, O. Vinyals, O. J. Hénaff, S. Albanie, Z. Akata, and M. Bethge, “A practitioner’s guide to real-world continual multimodal pretraining,” in Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, A. Globersons, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. M. Tomczak, and C. Zhang, Eds., 2024.
- [24] K. Nowakowski, M. Ptaszynski, K. Murasaki, and J. Nieuwazny, “Adapting multilingual speech representation model for a new, underresourced language through multilingual fine-tuning and continued pretraining,” Inf. Process. Manag., vol. 60, no. 2, 2023, p. 103148.
- [25] W. Zheng, W. Pan, X. Xu, L. Qin, L. Yue, and M. Zhou, “Breaking language barriers: Cross-lingual continual pre-training at scale,” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024, Y. Al-Onaizan, M. Bansal, and Y. Chen, Eds. Association for Computational Linguistics, 2024, pp. 7725–7738.
- [26] A. Cossu, A. Carta, L. C. Passaro, V. Lomonaco, T. Tuytelaars, and D. Bacciu, “Continual pre-training mitigates forgetting in language and vision,” Neural Networks, vol. 179, 2024, p. 106492.
- [27] M. DeHaven and J. Billa, “Improving low-resource speech recognition with pretrained speech models: Continued pretraining vs. semi-supervised training,” CoRR, vol. abs/2207.00659, 2022.
- [28] H. Zhu, G. Cheng, J. Wang, W. Hou, P. Zhang, and Y. Yan, “Boosting cross-domain speech recognition with self-supervision,” IEEE ACM Trans. Audio Speech Lang. Process., vol. 32, 2024, pp. 471–485.
- [29] N. San, G. Paraskevopoulos, A. Arora, X. He, P. Kaur, O. Adams, and D. Jurafsky, “Predicting positive transfer for improved low-resource speech recognition using acoustic pseudo-tokens,” in Proceedings of the 6th Workshop on Research in Computational Linguistic Typology and Multilingual NLP, SIGTYPE 2024, St. Julian’s, Malta, March 22, 2024, M. Hahn, A. Sorokin, R. Kumar, A. Scherbakov, Y. Otmakhova, J. Yang, O. Serikov, P. Rani, E. M. Ponti, S. Muradoglu et al., Eds. Association for Computational Linguistics, 2024, pp. 100–112.
- [30] A. A. Attia, D. Demszky, T. Ògúnrèmí, J. Liu, and C. Y. Espy-Wilson, “Cpt-boosted wav2vec2.0: Towards noise robust speech recognition for classroom environments,” CoRR, vol. abs/2409.14494, 2024.
- [31] M. Wortsman, G. Ilharco, S. Y. Gadre, R. Roelofs, R. G. Lopes, A. S. Morcos, H. Namkoong, A. Farhadi, Y. Carmon, S. Kornblith et al., “Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time,” in International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, ser. Proceedings of Machine Learning Research, K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvári, G. Niu, and S. Sabato, Eds., vol. 162. PMLR, 2022, pp. 23 965–23 998.
- [32] E. Yang, L. Shen, G. Guo, X. Wang, X. Cao, J. Zhang, and D. Tao, “Model merging in llms, mllms, and beyond: Methods, theories, applications and opportunities,” CoRR, vol. abs/2408.07666, 2024.
- [33] J. Choi, D. Kim, C. Lee, and S. Hong, “Revisiting weight averaging for model merging,” CoRR, vol. abs/2412.12153, 2024.
- [34] G. Ilharco, M. Wortsman, S. Y. Gadre, S. Song, H. Hajishirzi, S. Kornblith, A. Farhadi, and L. Schmidt, “Patching open-vocabulary models by interpolating weights,” in Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, Eds., 2022.
- [35] G. Stoica, D. Bolya, J. Bjorner, P. Ramesh, T. Hearn, and J. Hoffman, “Zipit! merging models from different tasks without training,” in The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
- [36] X. Jin, X. Ren, D. Preotiuc-Pietro, and P. Cheng, “Dataless knowledge fusion by merging weights of language models,” in The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
- [37] G. Ilharco, M. T. Ribeiro, M. Wortsman, L. Schmidt, H. Hajishirzi, and A. Farhadi, “Editing models with task arithmetic,” in The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
- [38] D. Li, Y. Ma, W. Wei, Q. Kong, Y. Wu, M. Che, F. Xia, E. Benetos, and W. Li, “Mertech: Instrument playing technique detection using self-supervised pretrained model with multi-task finetuning,” in IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2024, Seoul, Republic of Korea, April 14-19, 2024. IEEE, 2024, pp. 521–525.
- [39] E. Law, K. West, M. I. Mandel, M. Bay, and J. S. Downie, “Evaluation of algorithms using games: The case of music tagging,” in Proceedings of the 10th International Society for Music Information Retrieval Conference, ISMIR 2009, Kobe International Conference Center, Kobe, Japan, October 26-30, 2009, K. Hirata, G. Tzanetakis, and K. Yoshii, Eds. International Society for Music Information Retrieval, 2009, pp. 387–392.
- [40] M. Defferrard, K. Benzi, P. Vandergheynst, and X. Bresson, “FMA: A dataset for music analysis,” in Proceedings of the 18th International Society for Music Information Retrieval Conference, ISMIR 2017, Suzhou, China, October 23-27, 2017, S. J. Cunningham, Z. Duan, X. Hu, and D. Turnbull, Eds., 2017, pp. 316–323.
- [41] C. Papaioannou, I. Valiantzas, T. Giannakopoulos, M. A. Kaliakatsos-Papakostas, and A. Potamianos, “A dataset for greek traditional and folk music: Lyra,” in Proceedings of the 23rd International Society for Music Information Retrieval Conference, ISMIR 2022, Bengaluru, India, December 4-8, 2022, P. Rao, H. A. Murthy, A. Srinivasamurthy, R. M. Bittner, R. C. Repetto, M. Goto, X. Serra, and M. Miron, Eds., 2022, pp. 377–383.
- [42] X. Serra, “Creating research corpora for the computational study of music: the case of the compmusic project,” in AES International Conference on Semantic Audio 2014, London, UK, January 27-29, 2014, C. Dittmar, G. Fazekas, and S. Ewert, Eds. Audio Engineering Society, 2014.
- [43] B. Uyar, H. S. Atli, S. Sentürk, B. Bozkurt, and X. Serra, “A corpus for computational research of turkish makam music,” in Proceedings of the 1st International Workshop on Digital Libraries for Musicology, DLfM@JCDL 2014, London, United Kingdom, September 12, 2014, B. Fields and K. R. Page, Eds. ACM, 2014, pp. 1–7.
- [44] S. Sentürk, “Computational analysis of audio recordings and music scores for the description and discovery of ottoman-turkish makam music,” Ph.D. dissertation, Pompeu Fabra University, Spain, 2017.
- [45] A. Srinivasamurthy, G. K. Koduri, S. Gulati, V. Ishwar, and X. Serra, “Corpora for music information research in indian art music,” in Music Technology meets Philosophy - From Digital Echos to Virtual Ethos: Joint Proceedings of the 40th International Computer Music Conference, ICMC 2014, and the 11th Sound and Music Computing Conference, SMC 2014, Athens, Greece, September 14-20, 2014. Michigan Publishing, 2014.
- [46] C. Papaioannou, E. Benetos, and A. Potamianos, “From west to east: Who can understand the music of the others better?” in Proceedings of the 24th International Society for Music Information Retrieval Conference, ISMIR 2023, Milan, Italy, November 5-9, 2023, A. Sarti, F. Antonacci, M. Sandler, P. Bestagini, S. Dixon, B. Liang, G. Richard, and J. Pauwels, Eds., 2023, pp. 311–318.
- [47] C. Papaioannou, E. Benetos, and A. Potamianos, “LC-Protonets: Multi-label few-shot learning for world music audio tagging,” IEEE Open Journal of Signal Processing, vol. 6, 2025, pp. 138–146.
- [48] A. Défossez, J. Copet, G. Synnaeve, and Y. Adi, “High fidelity neural audio compression,” Trans. Mach. Learn. Res., vol. 2023, 2023.
- [49] W. Hsu, B. Bolte, Y. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,” IEEE ACM Trans. Audio Speech Lang. Process., vol. 29, 2021, pp. 3451–3460.
- [50] J. Kirkpatrick, R. Pascanu, N. C. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska et al., “Overcoming catastrophic forgetting in neural networks,” CoRR, vol. abs/1612.00796, 2016.
- [51] S. Hu, Y. Tu, X. Han, C. He, G. Cui, X. Long, Z. Zheng, Y. Fang, Y. Huang, W. Zhao et al., “Minicpm: Unveiling the potential of small language models with scalable training strategies,” CoRR, vol. abs/2404.06395, 2024.
- [52] Y. Guo, J. Fu, H. Zhang, D. Zhao, and Y. Shen, “Efficient continual pre-training by mitigating the stability gap,” CoRR, vol. abs/2406.14833, 2024.
- [53] M. D. Lange, G. M. van de Ven, and T. Tuytelaars, “Continual evaluation for lifelong learning: Identifying the stability gap,” in The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
- [54] I. A. P. Santana, F. Pinhelli, J. Donini, L. G. Catharin, R. B. Mangolin, Y. M. e Gomes da Costa, V. D. Feltrim, and M. A. Domingues, “Music4all: A new music database and its applications,” in 2020 International Conference on Systems, Signals and Image Processing, IWSSIP 2020, Niterói, Brazil, July 1-3, 2020. IEEE, 2020, pp. 399–404.
- [55] G. I. Parisi, R. Kemker, J. L. Part, C. Kanan, and S. Wermter, “Continual lifelong learning with neural networks: A review,” Neural Networks, vol. 113, 2019, pp. 54–71.
- [56] D. Kim and B. Han, “On the stability-plasticity dilemma of class-incremental learning,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023. IEEE, 2023, pp. 20 196–20 204.
- [57] R. Xiong, Y. Yang, D. He, K. Zheng, S. Zheng, C. Xing, H. Zhang, Y. Lan, L. Wang, and T. Liu, “On layer normalization in the transformer architecture,” in Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, ser. Proceedings of Machine Learning Research, vol. 119. PMLR, 2020, pp. 10 524–10 533.
- [58] R. Castellon, C. Donahue, and P. Liang, “Codified audio language modeling learns useful representations for music information retrieval,” in Proceedings of the 22nd International Society for Music Information Retrieval Conference, ISMIR 2021, Online, November 7-12, 2021, J. H. Lee, A. Lerch, Z. Duan, J. Nam, P. Rao, P. van Kranenburg, and A. Srinivasamurthy, Eds., 2021, pp. 88–96.
- [59] R. Yuan, Y. Ma, Y. Li, G. Zhang, X. Chen, H. Yin, L. Zhuo, Y. Liu, J. Huang, Z. Tian et al., “MARBLE: music audio representation benchmark for universal evaluation,” in Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, Eds., 2023.
- [60] Q. Huang, A. Jansen, J. Lee, R. Ganti, J. Y. Li, and D. P. W. Ellis, “Mulan: A joint embedding of music audio and natural language,” in Proceedings of the 23rd International Society for Music Information Retrieval Conference, ISMIR 2022, Bengaluru, India, December 4-8, 2022, P. Rao, H. A. Murthy, A. Srinivasamurthy, R. M. Bittner, R. C. Repetto, M. Goto, X. Serra, and M. Miron, Eds., 2022, pp. 559–566.
- [61] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
- [62] D. Karomat, “12 maqam system and its similarity with indian raga’s (according to the indian manuscripts),” Indian Musicological Society. Journal of the Indian Musicological Society, vol. 36, 2006, p. 62.
- [63] V. Blaschke, M. Fedzechkina, and M. ter Hoeve, “Analyzing the effect of linguistic similarity on cross-lingual transfer: Tasks and experimental setups matter,” CoRR, vol. abs/2501.14491, 2025.
- [64] E. Yang, Z. Wang, L. Shen, S. Liu, G. Guo, X. Wang, and D. Tao, “Adamerging: Adaptive model merging for multi-task learning,” in The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
- [65] M. Parovic, I. Vulic, and A. Korhonen, “Investigating the potential of task arithmetic for cross-lingual transfer,” in Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2024 - Volume 2: Short Papers, St. Julian’s, Malta, March 17-22, 2024, Y. Graham and M. Purver, Eds. Association for Computational Linguistics, 2024, pp. 124–137.