# JamendoMaxCaps: A Large Scale Music-caption Dataset with Imputed Metadata
**Authors**: Abhinaba Roy, Renhang Liu, Tongyu Lu, Dorien Herremans
> Singapore University of Technology and Design {abhnaba_roy, renhang_liu, tongyu_lu,
## Abstract
We introduce JamendoMaxCaps, a large-scale music-caption dataset featuring over 362,000 freely licensed instrumental tracks from the renowned Jamendo platform. The dataset includes captions generated by a state-of-the-art captioning model, enhanced with imputed metadata. We also introduce a retrieval system that leverages both musical features and metadata to identify similar songs, which are then used to fill in missing metadata using a local large language model (LLLM). This approach allows us to provide a more comprehensive and informative dataset for researchers working on music-language understanding tasks. We validate this approach quantitatively with five different measurements. By making the JamendoMaxCaps dataset publicly available, we provide a high-quality resource to advance research in music-language understanding tasks such as music retrieval, multimodal representation learning, and generative music models.
Index Terms: Music-caption dataset, music information retrieval, music retrieval, music representation learning, captioning, music generation, music AI.
## I Introduction
Music information retrieval is an active research field focused on developing computational techniques to analyze, organize, access, and even generate musical data. In recent years, rapid advances in deep learning and natural language processing have enabled novel approaches to music generation, where models can learn to generate music from textual descriptions [6, 20, 2] or other modalities [11]. However, the lack of large-scale, high-quality datasets with rich musical and textual annotations has been a key challenge in driving progress in these music-language understanding tasks.
In the sound/speech domain, we find several expansive datasets such as LAION-Audio [26] and wav2vec [18], each comprising over 500,000 audio-caption pairs. The music domain, however, is relatively limited in substantial publicly accessible resources. For instance, the widely utilized MusicCaps dataset [1] encompasses only approximately 5,000 examples. To address this scarcity, researchers frequently resort to data augmentation techniques to expand the dataset and generate synthetic captions based on the applied transformations [20]. Existing research has explored an approach that involves extracting musical features from pre-trained models and subsequently rephrasing them using a large language model, or even utilizing existing tags present in audio datasets in the form of metadata and using an LLM to generate captions based on them [7]. However, extracting individual features from a song can be a time-consuming process, and in practice, metadata is often incomplete or unavailable for many music collections.
<details>
<summary>extracted/6444849/jamendoxmax_overview_3.png Details</summary>
### Visual Description
## Flowchart: Music Captioning Pipeline
### Overview
The flowchart illustrates a technical pipeline for generating captions from music files. It begins with a data source (Jamendo Music), processes music files with partial metadata, imputes missing metadata, and generates captions using a music captioning model (Qwen2-Audio). The diagram includes visual elements like icons, color-coded status indicators, and directional arrows to represent data flow.
### Components/Axes
1. **Data Source**:
- Label: "Jamendo Music" (red hexagonal icon)
- Output: "Music File" (gray file icon with red musical note)
- Sub-component: "Partial Metadata" (orange magnifying glass with "X" symbol)
2. **Metadata Imputation**:
- Central processing block with blue hexagonal icon
- Inputs:
- "Complete Metadata" (green checkmark)
- "Captions" (pencil icon)
- Output: Connects to "Music Captioner"
3. **Music Captioner**:
- Label: "Qwen2-Audio" (blue geometric icon)
- Input: "Metadata Imputation" output
- Output: "Captions" (stacked gray files with pencil icons)
4. **Final Output**:
- "JamendoMaxCaps" (blue text at bottom-left)
- Contains:
- "Music File" (gray file)
- "Complete Metadata" (green checkmark)
- "Captions" (pencil icon)
### Flow Direction
- Left-to-right primary flow:
`Data Source → Music File → [Split] → Complete Metadata / Metadata Imputation → Music Captioner → Captions`
- Secondary feedback loop:
`Metadata Imputation → Music Captioner → Captions`
### Key Observations
1. **Data Completeness Handling**:
- Files with "Partial Metadata" (orange X) are routed through Metadata Imputation
- Files with "Complete Metadata" (green checkmark) bypass imputation
2. **Color Coding**:
- Red: Data Source (Jamendo Music)
- Blue: Music Captioner (Qwen2-Audio)
- Green: Complete Metadata status
- Orange: Partial Metadata status
3. **Dual Output Path**:
- Complete Metadata and Captions are generated in parallel from Music Files
4. **Circular Dependency**:
- Captions from Music Captioner feed back into Metadata Imputation
### Interpretation
This pipeline demonstrates a robust system for music captioning that:
1. **Handles Incomplete Data**: By separating files with partial metadata for imputation, the system ensures no data is discarded due to incomplete information
2. **Optimizes Processing**: Files with complete metadata are fast-tracked, improving efficiency
3. **Uses Specialized AI**: The Qwen2-Audio model suggests domain-specific optimization for music-to-text conversion
4. **Creates Dual Outputs**: Maintains both metadata and captions, valuable for searchability and accessibility
The system architecture implies a focus on accessibility and metadata enrichment, with particular attention to handling real-world data imperfections (partial metadata). The feedback loop between Metadata Imputation and Music Captioner suggests iterative refinement of caption quality based on metadata completeness.
</details>
Figure 1: Overview of our proposed pipeline for the creation of the dataset.
To mitigate these limitations, we introduce JamendoMaxCaps Dataset: https://huggingface.co/datasets/amaai-lab/JamendoMaxCaps Code: https://github.com/AMAAI-Lab/JamendoMaxCaps, a large-scale music-caption dataset comprising over 362,000 freely licensed instrumental tracks from the renowned Jamendo platform, a source of user-contributed music. We leverage a state-of-the-art music captioning model [4] to generate natural language descriptions for each track. To address the issue of incomplete metadata, we propose a multimodal metadata imputation approach that utilizes audio features as well a local large language model to expand the incomplete metadata. This allows us to populate fields such as genre, tempo, and variable tags like mood and instrumentation, providing a more comprehensive and informative dataset for researchers working on music-language understanding tasks.
We first construct a retrieval system that identifies similar songs based on the available musical and (partial) metadata features. We utilize the widely adopted MERT features to represent the musical content, and employ a sentence transformer to encode the partially available metadata. This query system combines both musical and metadata features, enabling efficient retrieval of similar songs based on their musical contents as well as associated (partial) metadata. We then use this retrieval system to retrieve the top 10 most similar songs with incomplete metadata in the dataset. We employ a local large language model to leverage in-context learning, where the model learns from the related examples to infer the missing metadata fields, such as genre, tempo, and variable tags like mood and instrumentation. The use of a local large language model is advantageous as it provides more control, flexibility, and cost-effectiveness compared to alternative approaches such as using a pre-trained GPT model hosted on a remote server. The selection of relevant in-context examples is critical, as the large language model can leverage these similar examples to infer better and predict the missing metadata fields. This contextual information is essential for the model to make well-informed predictions and populate the incomplete metadata in the dataset. We perform extensive evaluations to validate the choice of in-context examples as well as the quality of the imputed metadata, which is essential for the overall success and reliability of the dataset.
The key contributions of our work can be summarized as:
- We present JamendoMaxCaps, a large-scale music-caption dataset featuring over 362,000 freely-licensed instrumental tracks from the Jamendo platform, along with generated captions.
- We also develop a metadata imputation approach that utilizes a local large language model (LLLM) to infer and populate missing metadata fields, such as genre, tempo, and variable tags like mood and instrumentation, resulting in a more comprehensive dataset.
- We construct a retrieval system that integrates musical and metadata features to identify similar songs, enabling efficient contextualization for the large language model to infer the missing metadata.
## II Related Work
In this section, we discuss recent text-music datasets, metadata imputation research as well as music feature retrieval systems. MusicCaps [1], one of the most popular text-music datasets, contains human-annotated captions for a limited set of around 5,000 music samples. Despite the high-quality descriptions, its small scale is not sufficient for data-hungry deep learning models. MusicBench [20] further augments MusicCaps 11x and modifying the captions accordingly using an LLM. Still, it is limited to around 52k instances, each 10s in duration. The WavCaps dataset [19] expanded upon this concept by collecting over 400,000 audio-caption pairs from multiple online sources, employing heuristic filtering methods to ensure relevance. However, WaveCaps is more focused on generic audio rather than musical sounds. The recently released SongDescriber dataset [17], contains 1.1k human-written text captions of 706 songs. The authors intended this dataset to be used as an evaluation set for text-to-music systems. SunoCaps [5] contains 256 prompt-generated music samples, showing an alternative way creating text-music dataset. Finally, Noise2music [10] introduced the MuLaMCap dataset (MuLan-LaMDA Music Caption) which consists of 400k music-text pairs created by annotating AudioSet content with a data mining pipeline, but the dataset is not publicly available.
The recently released Midicaps dataset [21] contains over 178,000 music MIDI files with associated text captions. They used predefined feature extractors to get relevant features such as key, chords, and time signature, and used an LLM to generate captions. This dataset contains captions for MIDI music, however, not audio music files. The approach from [21] aligns with existing methods that integrate symbolic representations with large language models for caption generation. In the waveform domain, LP-MusicCaps [7] explored a tag-to-caption approach, where pre-existing metadata from audio libraries was used as input for a large language model, producing pseudo-captions to describe musical content. However, they relied on pre-existing metadata, which often exhibits notable incompleteness and inconsistencies in real-world scenarios. This limitation can hinder the comprehensive representation of musical content and potentially introduce biases in the generated captions.
Recent research has also explored metadata imputation to address incomplete or missing annotations in music datasets. For example, [13] proposed a collaborative filtering approach that combines content-based and user-generated metadata for automatic music tagging, leveraging deep learning to predict missing attributes based on musical embeddings. Similarly, [12] introduced a transformer-based model that infers missing genre and mood labels from audio spectrograms, highlighting the role of multimodal learning in metadata completion. These works align with our use of large language models for metadata imputation, although our method uniquely incorporates a retrieval-based in-context learning strategy to enhance imputation accuracy.
In addition, music feature retrieval systems have played a crucial role in improving dataset utility. CLAP [9] employed a contrastive learning approach for aligning music and text embeddings, enabling zero-shot music retrieval based on textual queries. MERT [15] provided a powerful audio feature extractor designed specifically for music representation learning. Both models demonstrate the effectiveness of using pre-trained embeddings for understanding musical content. Salmonn [25] is an integrated multiple-source audio features including Q-Former queries [14], Whisper features [23], BEATs features [3] to a generate music captions . Our approach builds upon these works by integrating retrieval-based metadata imputation with local large language models, aiming to ensure more contextually relevant annotations.
Despite the advancements made by these prior works, challenges remain in ensuring accurate, and scalable metadata imputation for large-scale music-caption datasets. The JamendoMaxCaps dataset aims to address these gaps by leveraging a hybrid approach that combines musical representation, retrieval-based contextualization, and in-context learning via locally hosted large language models.
## III Method
In this section, we detail our caption generation, retrieval system and metadata imputation approach for the JamendoMaxCaps dataset. Details regarding the dataset creation are in the next section.
### III-A Caption Generation
To generate high-quality textual descriptions for our large-scale dataset, we employ Qwen2-Audio [4], a state-of-the-art music understanding model that has demonstrated competitive performance relative to other approaches (e.g., Salmonn [25]). Our primary objective is to create informative captions focusing on instrumentation, genre, mood, rhythm, and potential listening scenarios as these descriptions serve as a crucial component of JamendoMaxCaps. To carry out caption generation we perform the following two steps:
#### Segmented Captioning for Temporal Variation
In order to capture potential temporal changes throughout each track, we segmented every song into consecutive 30-second clips, generating a caption for each segment. We discarded any final segment shorter than 15 seconds as we found that the caption tends to be of lower quality when the given tracks are too short. While multiple captions per track provide richer training data for music understanding and generation models, for subsequent retrieval and metadata imputation tasks, we focused on using only the first 30-second segment as the track’s representative excerpt. This decision balances computational efficiency with capturing core musical characteristics, such as initial instrumentation, tempo, and mood.
#### Efficient Inference and Prompt Design
To manage the computational demands of captioning over 362,000 tracks, we employed batched inference and leveraged 4-bit quantization to reduce model memory requirements, with minimal impact on caption quality. We set a maximum output length of 128 tokens per caption, ensuring that roughly 90% of the generated text falls within this range, thus avoiding overly long or unwieldy outputs. Each 30-second clip was processed using a standardized prompt:
“ Generate a detailed caption for this music piece, primarily focusing on instrumentation, genre, mood, rhythm, and scenario. ”
Preliminary experiments indicate that explicitly mentioning both instrumentation and mood in the prompt yielded more coherent, contextually rich captions.
Our caption generation pipeline proves to be robust at scale, ultimately producing captions for more than 3 million 30-second audio segments, providing a reliable basis for subsequent metadata imputation and retrieval systems, as well as for broader music-language modeling efforts.
### III-B Retrieval System
To impute missing metadata fields, we first construct a retrieval system that identifies similar songs based on both musical and metadata features. For the musical content, we utilize MERT features [15] We use https://huggingface.co/m-a-p/MERT-v1-330M to represent the audio signals. We choose MERT due to its proven ability to understand as well as capture the musical content in a song [15], whereas models like CLAP are more geared towards general audio sounds.
To calculate features, we take the average of all layers, which provides a compact representation of the entire track. Mathematically, for a song $S$ with $T$ frames, the resulting final MERT feature vector is calculated as:
$$
M(S)=\frac{1}{T}∑_t=1^TM_n(S_t);n∈\{1...N\} \tag{1}
$$
where, $M_n$ is the MERT latent features at the $n$ -th layer and $N$ is the total number of layers (25 in our case) in the MERT feature extractor. This results in a $N×$ 1024-dimensional vector representation of the musical content for each song, i.e., $M(S)∈ℝ^N× 1024$ .
For the metadata features, as the dataset contains partially available metadata, we first encode the existing metadata fields using the Google flan-t5 model https://huggingface.co/google/flan-t5-small. This sentence transformer encodes the metadata fields into a 768-dimensional vector. Mathematically, for a song $S$ with metadata $T$ , the encoded metadata feature vector is: $F(S)$ , with $F∈ℝ^768$ .
To retrieve the most similar songs to a given song $S$ , we concatenate the musical and metadata feature vectors and compute the cosine similarity between the query song and all other songs in the dataset. Since the dimensions of the MERT audio features and the metadata features are different, we align the MERT features using sparse random projection – a dimensionality reduction technique to match the 768 dimensions of the metadata embeddings. Mathematically,
$$
X(S)=λ_1×(P(M(\mathcal{S)
})+λ_2×F(\mathcal{S)} \tag{2}
$$
Where $P(.)$ is the sparse random projection function, $λ_1$ and $λ_2$ are hyperparameters to balance the contribution of audio and metadata features such that $λ_1+λ_2=1$ . Here, $X∈ℝ^768$ . The similarity score between a query song $Q$ and a song $S$ in the dataset is calculated as:
$$
Similarity(Q,S)=\frac{X(Q)^
⊤X(S)}{||X(Q)||×||X
(S)||} \tag{3}
$$
Using this retrieval system, we can identify songs that are similar in both musical and metadata features to the query song. For each song in our dataset, we retrieve the top 10 most similar songs based on this similarity metric. We use the (partial) metadata and captions generated from the audio of these similar songs as the in-context examples for the large language model to impute the missing metadata fields, as described in the next section.
### III-C Metadata Imputation
To impute the missing metadata fields, we employ the capabilities of large language models. We have opted for a locally hosted large language model to maintain full control over the system and avoid dependence on remote servers. Specifically, we utilize Llama-2, a large language model with 7 billion parameters https://huggingface.co/meta-llama/Llama-2-7b-hf, which is hosted on our own infrastructure.
<details>
<summary>extracted/6444849/meta_imputation.png Details</summary>
### Visual Description
## Flowchart: Metadata Processing System
### Overview
The flowchart illustrates a two-stage system for processing musical data and generating enriched metadata. It consists of an Input section, two processing components (Custom Retrieval System and Large Language Model), and an Output section. Arrows indicate data flow between components.
### Components/Axes
**Input Section (Blue Box):**
- **Musical Data** (Top)
- **Partial Metadata** (Middle)
- **Captions** (Bottom)
**Processing Components:**
1. **Custom Retrieval System** (Gray Box)
- Receives input from all three Input components
- Outputs to "Retrieved Similar Songs" (Orange Box)
2. **Large Language Model** (Green Box)
- Receives input from all three Input components
- Outputs to "Completed Metadata" (Light Green Box)
**Output Section (Orange Box):**
- **Retrieved Similar Songs** (Top)
- **Completed Metadata** (Bottom)
**Arrows:**
- Black arrows connect Input components to processing systems
- Arrows from processing systems to Output components
- Colors: Blue (Input), Gray (Custom Retrieval), Green (Language Model), Orange (Output)
### Detailed Analysis
**Input Components:**
1. Musical Data: Represented by a white rectangle with black text
2. Partial Metadata: Pink rectangle with black text
3. Captions: Yellow rectangle with black text
**Processing Flow:**
1. All Input components feed into both processing systems
2. Custom Retrieval System processes data to identify similar songs
3. Large Language Model enriches metadata through language understanding
**Output Components:**
1. Retrieved Similar Songs: Orange box with stacked pink rectangles
2. Completed Metadata: Light green box with black text
### Key Observations
1. Bidirectional processing: Both systems receive identical input data
2. Output specialization: Different systems handle song retrieval vs metadata completion
3. Color coding: Clear visual distinction between Input (blue), processing (gray/green), and Output (orange/light green)
### Interpretation
This flowchart represents a hybrid system combining:
1. **Retrieval-based processing** (Custom Retrieval System) for finding similar content
2. **Generative language processing** (Large Language Model) for metadata enrichment
The system demonstrates:
- **Data fusion**: Multiple input types (audio metadata + text captions) are combined
- **Parallel processing**: Both systems operate simultaneously on the same input
- **Output differentiation**: Separate handling of content discovery vs metadata completion
The color-coded structure suggests a modular architecture where:
- Input components are foundational data sources
- Processing systems represent distinct AI/ML capabilities
- Output components represent actionable results for music recommendation systems
No numerical data or trends are present - this is a conceptual architecture diagram rather than a data visualization.
</details>
Figure 2: The metadata imputation process.
Algorithm 1 shows the pipeline for metadata imputation. For each song in the JamendoMaxCaps, we first retrieve the top 10 most similar songs using the retrieval system described earlier. We then construct an in-context examples by concatenating the metadata and captions of these top 10 similar songs. In-context learning has proven to be highly effective in empowering large language models to generate meaningful and relevant outputs [8]. We leverage this approach by prompting the language model with these in-context examples and provide the caption of a target song, asking the large language model to infer the corresponding metadata https://github.com/AMAAI-Lab/JamendoMaxCaps. This process allows the language model to draw insights from the contextual information of similar songs to impute the missing metadata fields for the target song.
Algorithm 1 Imputing Missing Metadata
1: for each song $Q∈{{JamendoMaxCaps}}$ do
2: Step 1: Retrieve Similar Songs Retrieve the top $k$ similar songs, denoted as:
$$
\{S_1,S_2,\dots,S_k\}
$$
using the retrieval system.
3: Step 2: Construct In-Context Example Formulate the in-context example $E$ as:
$$
E=\Big{(}(M(S_i),C(S_i))
\big{|} S_i∈\{S_1,S_2,\dots,
S_k\}\Big{)}
$$
Note: $M(S_i)$ – metadata; $C(S_i)$ – caption of song $S_i$ .
4: Step 3: Prompt Language Model Use the language model $L$ with $E$ and the caption of the target song $Q$ , denoted as $C(Q)$ :
$$
M(Q)=L(E,C(Q))
$$
5: Step 4: Infer Metadata Infer the missing metadata fields $M(Q)$ for song $Q$ .
6: end for
## IV Evaluation And Statistics
Here, we first discuss the dataset creation process, and then detail the results of the metadata imputation.
### IV-A JamendoMaxCaps Dataset
To create the dataset, we first collected audio tracks and associated metadata from the Jamendo music platform, a widely used online repository of Creative Commons-licensed music. We used their publicly accessible API to download instrumental songs that were uploaded to the platform between 2008 and 2023 (both years included), downloading the raw audio tracks and extracting the corresponding metadata fields. This process yielded a total of 362,649 audio tracks with partial metadata. We then cleaned the dataset by removing tracks with very short durations, resulting in a final set of 362,238 tracks with an average duration of 4 minutes and 12 seconds. A brief overview of some of the properties of the dataset is presented in Table I.
Next, we use the Qwen2-Audio [4] to generate a caption that describes the musical content of each track in natural language. We take the caption of the first 30 seconds( III-A) for computational efficiency. This allows us to pair each audio track with a descriptive caption.
Finally, we apply the metadata imputation approach discussed in the previous section. We observe that since all songs in our dataset are instrumental, ‘lang’ (language) and ‘gender’ (gender of singer) are always empty. We discard these for the evaluation of the imputation task. Considering only the rest of the metadata fields, 261,935 songs had partial metadata, while the remaining 100,303 tracks had complete metadata information. We focus on imputing the missing metadata fields for the songs with partial metadata. To create the final dataset, we concatenated the songs, their corresponding captions, and the (imputed where applicable) metadata information. A couple of examples of metadata imputation are given in Table II.
TABLE I: Statistical Summary of the Dataset
| Length (seconds) a SNR (dB) b RMS Energy b | $252.30$ $0.03$ $0.15$ | $209.853$ $0.001$ $0.139$ | $124.959$ $-0.01$ $0.117$ |
| --- | --- | --- | --- |
| Zero Crossing Rate b | $0.047$ | $0.041$ | $0.029$ |
a Dataset has 362,238 examples. b SNR = Signal-to-noise ratio; RMS Energy = Signal’s overall power; Zero Crossing Rate = How often a signal crosses the zero amplitude level, indicating the rate of sign changes.
TABLE II: Comparison of Metadata Before and After Imputation a
| ‘vocalinstrumental’: ‘instrumental’, ‘lang’: ‘’, ‘gender’: ‘’, ‘speed’: ‘medium’, ‘tags’: ‘genres’: [], ‘instruments’: [], ‘vartags’: [] | ‘vocalinstrumental’: ‘instrumental’, ‘lang’: ‘’, ‘gender’: ‘’, ‘speed’: ‘medium’, ‘tags’: ‘genres’: [‘ pop ’, ‘ electronic ’, ‘ hip-hop ’], ‘instruments’: [‘ bass ’, ‘ drums ’], ‘vartags’: [‘ intense ’, ‘ energetic ’] |
| --- | --- |
| ‘vocalinstrumental’: ‘instrumental’, ‘lang’: ”, ‘gender’: ”, ‘speed’: ’medium’, ‘tags’: ‘genres’: [], ‘instruments’: [], ‘vartags’: [] | ‘vocalinstrumental’: ‘instrumental’, ‘lang’: ”, ‘gender’: ”, ‘speed’: ’medium’, ‘tags’: ‘genres’: [‘ cyberpunk ’], ‘instruments’: [], ‘vartags’: [‘ synthwave ’] |
| a Bold keys represent the metadata fields. Green text highlights newly imputed values. | |
<details>
<summary>extracted/6444849/genres_plots.png Details</summary>
### Visual Description
## Bar Charts: Top Original vs. Top Imputed Genres
### Overview
The image contains two side-by-side bar charts comparing "Top Original Genres" (left) and "Top Imputed Genres" (right). Both charts use vertical bars with distinct color coding for genres. The y-axis represents numerical values, while the x-axis lists genre categories. The charts emphasize differences in genre popularity or representation between original and imputed datasets.
---
### Components/Axes
#### Left Chart (Top Original Genres):
- **X-axis**: Genres (categorical):
`electronic`, `ambient`, `rock`, `filmscore`, `experimental`, `classical`, `hiphop`, `pop`, `dance`, `corporate`
- **Y-axis**: Numerical values (0–14,000 range, increments of 2,000).
- **Colors**:
- `electronic`: Pink (#FF69B4)
- `ambient`: Orange (#FF8C00)
- `rock`: Olive (#6B8E23)
- `filmscore`: Forest green (#228B22)
- `experimental`: Teal (#008080)
- `classical`: Turquoise (#40E0D0)
- `hiphop`: Blue (#0000FF)
- `pop`: Light blue (#ADD8E6)
- `dance`: Purple (#9370DB)
- `corporate`: Magenta (#FF00FF)
#### Right Chart (Top Imputed Genres):
- **X-axis**: Genres (categorical):
`electronic`, `ambient`, `techno`, `rock`, `classical`, `experimental`, `pop`, `jazz`, `soundtrack`, `dance`
- **Y-axis**: Numerical values (0–45,000 range, increments of 10,000).
- **Colors**:
- `electronic`: Pink (#FF69B4)
- `ambient`: Orange (#FF8C00)
- `techno`: Olive (#6B8E23)
- `rock`: Forest green (#228B22)
- `classical`: Teal (#008080)
- `experimental`: Turquoise (#40E0D0)
- `pop`: Blue (#0000FF)
- `jazz`: Light blue (#ADD8E6)
- `soundtrack`: Purple (#9370DB)
- `dance`: Magenta (#FF00FF)
---
### Detailed Analysis
#### Left Chart (Top Original Genres):
- **Highest Value**: `electronic` (14,074).
- **Second-Highest**: `ambient` (7,231).
- **Trend**: Values decrease monotonically from left to right:
`electronic` (14,074) > `ambient` (7,231) > `rock` (5,653) > `filmscore` (5,524) > `experimental` (3,822) > `classical` (3,730) > `hiphop` (2,540) > `pop` (2,517) > `dance` (2,368) > `corporate` (2,188).
#### Right Chart (Top Imputed Genres):
- **Highest Value**: `electronic` (44,540).
- **Second-Highest**: `ambient` (17,328).
- **Trend**: Values decrease monotonically from left to right:
`electronic` (44,540) > `ambient` (17,328) > `techno` (15,456) > `rock` (13,255) > `classical` (11,361) > `experimental` (9,665) > `pop` (9,509) > `jazz` (8,833) > `soundtrack` (8,278) > `dance` (5,789).
---
### Key Observations
1. **Genre Shifts**:
- The imputed chart replaces `filmscore`, `hiphop`, `pop`, and `corporate` (from the original chart) with `techno`, `jazz`, and `soundtrack`.
- `dance` appears in both charts but with significantly lower values in the imputed version (5,789 vs. 2,368).
2. **Value Discrepancies**:
- `electronic` and `ambient` show the largest absolute increases in the imputed chart (+30,466 and +10,097, respectively).
- `dance` is the only genre with a decrease in the imputed chart (-1,781).
3. **Color Consistency**:
- Colors for shared genres (`electronic`, `ambient`, `rock`, `experimental`, `classical`) match between charts.
- New genres in the imputed chart (`techno`, `jazz`, `soundtrack`) use colors from the original chart’s unused categories (`filmscore`, `hiphop`, `pop`).
---
### Interpretation
1. **Data Source Implications**:
- The imputed chart’s higher values (e.g., `electronic` at 44,540 vs. 14,074) suggest the imputation process either:
- Corrected underrepresented data in the original dataset.
- Introduced synthetic data to fill gaps, potentially inflating totals.
2. **Genre Evolution**:
- The replacement of `filmscore`/`hiphop`/`pop` with `techno`/`jazz`/`soundtrack` in the imputed chart may reflect:
- Updated genre classifications.
- Algorithmic bias in imputation favoring certain genres.
3. **Anomalies**:
- `dance`’s value drops in the imputed chart despite being retained, indicating potential inconsistencies in the imputation logic.
- The absence of `corporate` in the imputed chart suggests it was deemed irrelevant or excluded during processing.
4. **Structural Differences**:
- The imputed chart has 10 genres (same as original), but the x-axis order differs, prioritizing `techno` and `jazz` over `filmscore`/`hiphop`.
---
### Conclusion
The charts highlight significant differences in genre representation between original and imputed datasets. The imputed chart’s higher values and altered genre composition suggest either data correction or algorithmic bias. Further investigation into the imputation methodology is warranted to validate these findings.
</details>
Figure 3: Distribution of genres in original and imputed metadata.
<details>
<summary>extracted/6444849/speed_plots.png Details</summary>
### Visual Description
## Bar Charts: Top Original Speed vs. Top Imputed Speed
### Overview
The image contains two side-by-side bar charts comparing "Top Original Speed" (left) and "Top Imputed Speed" (right). Both charts use vertical bars to represent numerical values across categorical speed classifications. The left chart has 5 categories, while the right chart has 10 categories, suggesting a more granular breakdown in the imputed data.
### Components/Axes
**Left Chart (Top Original Speed):**
- **X-axis (Categories):** medium, high, low, very low, very high
- **Y-axis (Values):** 0 to 80,000 (linear scale)
- **Colors:**
- Pink (#FF69B4) for "medium"
- Orange (#FF8C00) for "high"
- Green (#8B4513) for "low"
- Teal (#30D5C8) for "very low"
- Blue (#0000FF) for "very high"
**Right Chart (Top Imputed Speed):**
- **X-axis (Categories):** medium, high, low, fast, slow, very low, very high, very fast, moderate
- **Y-axis (Values):** 0 to 80,000 (linear scale)
- **Colors:**
- Pink (#FF69B4) for "medium"
- Orange (#FF8C00) for "high"
- Green (#8B4513) for "low"
- Teal (#30D5C8) for "fast"
- Blue (#0000FF) for "slow"
- Purple (#800080) for "very low"
- Magenta (#FF00FF) for "very high"
- Cyan (#00FFFF) for "very fast"
- Light Purple (#E6E6FA) for "very fast"
- Dark Blue (#00008B) for "moderate"
### Detailed Analysis
**Left Chart (Top Original Speed):**
- **medium:** 77,614 (tallest bar)
- **high:** 14,218
- **low:** 8,581
- **very low:** 3,487
- **very high:** 2,928
**Right Chart (Top Imputed Speed):**
- **medium:** 77,082
- **high:** 14,046
- **low:** 5,894
- **fast:** 4,729
- **slow:** 4,534
- **very low:** 2,151
- **very high:** 1,927
- **very fast:** 1,204
- **moderate:** 72
- **additional category:** 39 (unlabeled)
### Key Observations
1. **Dominant Category:** "Medium" speed dominates both datasets, accounting for ~99% of total values in both charts.
2. **Categorization Differences:** The imputed data (right chart) introduces new categories ("fast," "slow," "very fast," "moderate") with significantly lower values.
3. **Value Trends:**
- Original data shows a gradual decline from "medium" to "very high" (77,614 → 2,928).
- Imputed data shows a steeper decline across more categories (77,082 → 39).
4. **Outliers:** The "moderate" category in the right chart has an extremely low value (39), suggesting potential data quality issues or rare edge cases.
### Interpretation
The data suggests that:
1. **Original Data Granularity:** The original speed classifications (left chart) use broader categories, with "medium" being overwhelmingly dominant.
2. **Imputation Refinement:** The imputed data (right chart) introduces finer-grained categories, revealing previously undetected speed variations but with much smaller magnitudes.
3. **Data Quality Concerns:** The extreme outlier in the "moderate" category (39) warrants investigation - this could indicate:
- Data entry errors
- Rare edge cases in the dataset
- Imputation algorithm limitations
4. **Distribution Patterns:** Both charts show a clear power-law distribution, with the majority of data concentrated in the "medium" category and rapid decay in extreme speed classifications.
The charts collectively demonstrate how data imputation can reveal hidden patterns in categorical data, but also highlight the challenges of interpreting imputed values, particularly when dealing with extreme outliers or rare categories.
</details>
Figure 4: Distribution of speed in original and imputed metadata.
<details>
<summary>extracted/6444849/vartags_plots.png Details</summary>
### Visual Description
## Bar Charts: Top Original vs. Top Imputed Vartags
### Overview
The image contains two side-by-side bar charts comparing "Top Original Vartags" (left) and "Top Imputed Vartags" (right). Both charts display categorical data with numerical values, using distinct color-coded bars for each category. The original chart focuses on emotional/psychological states, while the imputed chart emphasizes mood and atmospheric themes.
### Components/Axes
#### Left Chart (Top Original Vartags)
- **X-axis**: Categories (emotional/psychological states)
- peaceful, energetic, instrumental, happy, dark, upbeat, atmospheric, inspirational, film, emotional
- **Y-axis**: Numerical values (0–3,500)
- **Legend**: Color-coded categories (pink, orange, green, teal, blue, purple)
- **Bar Colors**:
- peaceful (pink), energetic (orange), instrumental (green), happy (green), dark (teal), upbeat (teal), atmospheric (blue), inspirational (blue), film (purple), emotional (purple)
#### Right Chart (Top Imputed Vartags)
- **X-axis**: Categories (mood/atmospheric themes)
- relaxing, dark, atmospheric, energetic, happy, futuristic, calm, peaceful, sad, dreamy
- **Y-axis**: Numerical values (0–5,000)
- **Legend**: Color-coded categories (pink, orange, green, brown, teal, blue, purple)
- **Bar Colors**:
- relaxing (pink), dark (orange), atmospheric (green), energetic (brown), happy (green), futuristic (teal), calm (blue), peaceful (light blue), sad (purple), dreamy (pink)
### Detailed Analysis
#### Left Chart (Top Original Vartags)
- **Highest Value**: "peaceful" (3,519, pink)
- **Lowest Value**: "emotional" (1,070, purple)
- **Trend**: Values decrease from left to right, with a sharp drop after "dark" (1,649) to "atmospheric" (1,583) and further declines to "inspirational" (1,125) and "film" (1,096).
#### Right Chart (Top Imputed Vartags)
- **Highest Value**: "relaxing" (5,272, pink)
- **Lowest Value**: "dreamy" (3,995, pink)
- **Trend**: Values decrease from left to right, with a gradual decline from "dark" (4,942) to "dreamy" (3,995). "Calm" (4,157) and "peaceful" (4,103) show near-identical values.
### Key Observations
1. **Value Discrepancies**:
- Imputed values are consistently higher than original values for overlapping categories (e.g., "dark" increases from 1,649 to 4,942; "happy" from 1,733 to 4,686).
- "Peaceful" appears in both charts but with vastly different values (3,519 vs. 4,103).
2. **Category Overlap**:
- "Dark," "happy," and "peaceful" appear in both charts but with divergent values.
- Unique categories: "instrumental," "upbeat," "inspirational," "film," "emotional" (original); "relaxing," "atmospheric," "energetic," "futuristic," "calm," "sad," "dreamy" (imputed).
3. **Color Consistency**:
- "Dark" (teal in original, orange in imputed) and "happy" (green in both) retain consistent colors across charts.
- "Peaceful" shifts from pink (original) to light blue (imputed), suggesting distinct categorization.
### Interpretation
The data suggests that the imputation process significantly alters the distribution and magnitude of Vartag values. Imputed categories like "relaxing" and "dark" dominate with higher values, potentially indicating that the imputation method prioritizes these themes or fills missing data with estimates skewed toward these categories. The original chart’s lower values and narrower category range imply that the original dataset may have been more limited in scope or resolution. The divergence in "peaceful" values (3,519 vs. 4,103) highlights inconsistencies between original and imputed data, raising questions about the imputation algorithm’s reliability or the original data’s completeness. The overlapping categories with divergent values (e.g., "dark," "happy") suggest that imputation may introduce biases or amplify certain themes, warranting further investigation into the methodology’s impact on data integrity.
</details>
Figure 5: Distribution of variable tags in original and imputed metadata.
Furthermore, we provide a summary of the distribution of some key metadata fields in the dataset to offer additional insight into its composition. This includes an analysis of the distribution of genres, and speed characteristics of the songs, which can help researchers and practitioners better understand the dataset and its potential applications. Figure 3 shows that the metadata imputation process adds more examples of previously underrepresented categories, such as “Techno” and “Jazz,” while also retaining the representation of major genres “Electronic” and “Rock”. This imputation effectively fills in the missing metadata fields, resulting in a more comprehensive and informative dataset representation. Moreover, the prevalence of “Electronic” and “Rock” genres in the dataset suggests the inclusion of more contemporary musical works, which aligns with the time period of our data collection efforts (2008-2023). In the case of speed, the imputation process enhances the granularity of the metadata by introducing additional categories, including “Fast”, “Slow”, “Very Fast”, and “Moderate”, which were not initially present significantly in the original data. While “Medium” remains the predominant category, the imputed metadata offers a more nuanced distribution, capturing a broader range of tempo variations. Noteworthy, in the original data, missing/nothing was the third most common category, but the imputation process successfully eliminated this gap, adding valuable information to the dataset. Figure 5 illustrates a significant enhancement in the diversity and coverage of vartags, providing more representation of musical characteristics. While the original data concentrated on a few prevalent tags such as “peaceful” and “energetic,” the imputed metadata introduces a wider range of descriptors, including “relaxing,” “futuristic,” and “dreamy”, which better encapsulate the emotional and atmospheric elements of the music.
In summary, the JamendoMaxCaps dataset, created through a combination of data collection, caption generation, and metadata imputation, offers a comprehensive and diverse music repository with rich descriptive information.
TABLE III: Subjective evaluation results for different retrieval configurations
| $λ_1=0$ ; $λ_2=1$ (Only metadata) | 2.0 | 3.0 |
| --- | --- | --- |
| $λ_1=1$ ; $λ_2=0$ (Only musical features) | 3.0 | 3.0 |
| $λ_1=0.4$ ; $λ_2=0.6$ (Metadata favored) | 4.0 | 4.0 |
| $λ_1=0.6$ ; $λ_2=0.4$ (Musical features favored) | 6.0 | 6.0 |
The ratings represent the scores given by two independent participants on a 7-point Likert scale [16] – from 1 (very bad) to 7 (very good).
TABLE IV: Evaluation of Metadata Imputation with Generic Examples vs Retrieval Examples
| Evaluation Metric BERT-Score $↑$ | Imputation with Generic Examples Genres 0.51 | Imputation with Retrieval Examples Speed 0.70 | Vartags 0.35 | Genres 0.58 | Speed 0.73 | Vartags 0.38 |
| --- | --- | --- | --- | --- | --- | --- |
| BLEU-1 $↑$ | 0.17 | 0.61 | 0.02 | 0.19 | 0.64 | 0.03 |
| BLEU-2 $↑$ | 0.07 | 0.58 | 0.01 | 0.08 | 0.60 | 0.01 |
| BLEU-3 $↑$ | 0.05 | 0.57 | 0.01 | 0.06 | 0.60 | 0.02 |
| BLEU-4 $↑$ | 0.05 | 0.57 | 0.01 | 0.05 | 0.60 | 0.01 |
### IV-B Retrieval System Settings
Since Qwen2-Audio [4] is an established model, we do not evaluate its performance. Instead, we focus on the strength of the retrieval system and the metadata imputation approach.
To evaluate the quality of the retrieval system used for metadata imputation, we explore different combinations of musical features and metadata features as outlined in Equation 2. We vary the values of $λ_1$ and $λ_2$ in four distinct ways:
- $λ_1=0$ ; $λ_2=1$ – Only rely on metadata information
- $λ_1=1$ ; $λ_2=0$ – Only rely on musical features
- $λ_1=0.4$ ; $λ_2=0.6$ – Favour metadata features more over musical features
- $λ_1=0.6$ ; $λ_2=0.4$ – Favour musical features more over metadata features
We conduct a small-scale subjective evaluation involving two independent evaluators to assess the relevance and similarity of the retrieved songs. The participants were asked to rate the overall quality of the retrieval on a 7-point Likert scale [16] – 1 being least similar, 7 being most similar songs. From the dataset, we select five songs and retrieve similar songs based on the four variations of $λ_1$ and $λ_2$ mentioned above. The results are presented in Table III, which shows that the combination of $λ_1=0.6$ and $λ_2=0.4$ outperformed the other three options. Looking at the stark difference in the performance, we decided not to carry out any further subjective evaluation (with more participants) related to $λ$ .
### IV-C Objective Evaluation
To ensure the reliability of our metadata imputation approach and to verify that the local language model we use does not hallucinate, we conduct two-fold objective evaluations.
To evaluate the accuracy of the metadata imputation, we first identify the examples that have valid entries in the original metadata for the ‘genres’, ‘speed’, and ‘vartags’ fields. We then generate imputed values for these fields using our approach and compare them to the original values for 5,000 examples per field. As the primary metric, we employ BERT-Score [27] to measure the semantic similarity between the original and imputed values. Since the imputation can generate multiple predictions, this allows us to establish a semantic similarity score between the original and imputed data. Additionally, we utilize n-gram metrics such as BLEU1 to BLEU4 (B1, B2, B3, B4) [22] to assess the exact overlap between the original and imputed values. Please note that the primary purpose of BLEU scores is to compare machine translations with ground truth, rather than evaluating semantic similarity.
Next, to evaluate the effectiveness of in-context learning for metadata imputation, we generate a secondary baseline that learns from a set of 10 random examples during the in-context learning phase, rather than utilizing the most similar songs retrieved through our retrieval system. This approach allows us to assess the impact of leveraging contextually relevant information versus a generic set of examples when imputing missing metadata fields. We also compute the BERT-Score and BLEU metrics (BLEU1 through BLEU4) to evaluate the semantic similarity and n-gram overlap between the original and imputed metadata values, respectively. This evaluation is on the same 5,000 examples per metadata field, as discussed before. The results are compiled in Table IV. Results demonstrate that metadata imputation using retrieval examples consistently outperforms imputation with generic examples across all evaluation metrics. Specifically, the BERT-Score is higher for all fields when retrieval examples are used, indicating a better semantic alignment between the original and imputed metadata. The relatively high BERT-Scores (e.g., 0.58 for genres, 0.73 for speed, when using retrieval examples) suggest that the imputation method is producing meaningful values that closely resemble the original metadata. On the other hand, the BLEU scores, especially for genres and vartags, are quite low, which reflects the nature of the imputation task. These low scores (e.g., 0.05 for genres and 0.01 for vartags) highlight the challenge of achieving perfect overlap with the original values, as metadata fields like genres and vartags often contain a broad range of possible terms, making exact matches unlikely. In contrast, the higher BLEU scores for speed (e.g., 0.64 for retrieval examples) can be attributed to the limited and more constrained set of values (e.g., ‘fast’, ‘slow’), which results in a higher likelihood of overlap. Overall, these results confirm that our metadata imputation approach, which leverages contextually relevant retrieval examples, is effective in generating meaningful and accurate imputed values.
TABLE V: Results of the listening study evaluating imputed metadata
| Human-like Generation Genres Matching Speed Matching | 5.32 $±$ 0.31 5.09 $±$ 0.69 5.82 $±$ 0.43 |
| --- | --- |
| Vartags Matching | 5.59 $±$ 0.45 |
### IV-D Subjective Evaluation
To subjectively verify the quality of the imputation, we conduct a listening study using the PsyToolkit platform https://www.psytoolkit.org/ [24]. Study participants are asked to listen to 10 randomly selected audio files and rate the associated imputed metadata on four criteria: 1) the extent to which the metadata could have been generated by a human, 2) the accuracy of the imputed genres, 3) the accuracy of the imputed speed, and 4) the accuracy of the imputed vartags. A 7-point Likert scale [16] is used for scoring, with 1 being the lowest and 7 being the highest. A total of 15 participants took part in the listening study. Questions and samples: https://github.com/AMAAI-Lab/JamendoMaxCaps.
The results of the listening study are presented in Table V. We achieve consistently high scores, with all metrics rated above 5.09 out of 7, indicating a strong overall performance. ‘speed’ and ‘vartags’ matching received particularly impressive scores of 5.82 and 5.59 respectively, with low standard deviations (0.43, 0.45) suggesting reliable and consistent performance across different musical pieces. The high score for human-like generation (5.32 $±$ 0.31) indicates that our system successfully captures the characteristics that human annotators would typically provide. Even the relatively lower score for ‘genres’ matching (5.09 $±$ 0.69) remains above the median point of the scale, demonstrating competent genre classification capabilities. These results suggest that our approach could serve as a reliable tool for automatic metadata generation.
## V Discussion
While our evaluation confirms the effectiveness of the JamendoMaxCaps dataset in improving metadata completeness and retrieval quality, several challenges and areas for future refinement remain. One primary challenges is the reliance on imputed metadata, which, despite leveraging a local large language model (LLLM) and retrieval-based contextualization, remains inherently constrained by the quality and representativeness of the retrieved examples. As demonstrated in our evaluation, the imputed metadata fields, particularly those related to genre and variable tags (e.g., mood descriptors), exhibit lower BLEU scores, suggesting a notable degree of variability in how different musical elements are categorized. This variability arises from the inherent subjectivity in musical taxonomy, where the same piece may be classified under different labels depending on context and personal interpretation. Our approach does not entirely eliminate inconsistencies, especially for songs with highly unique stylistic elements that may not align well with the retrieved examples.
Another limitation comes from potential biases in the retrieval system. The dataset composition influences the quality of metadata imputation, as genres with a higher representation, such as “Electronic” and “Rock”, benefit from a richer pool of similar examples, whereas underrepresented genres, such as “Jazz” or “Experimental”, may suffer from less reliable imputations due to the scarcity of relevant contextual data. This imbalance can lead to an over-representation of dominant categories in the imputed metadata, potentially reinforcing dataset biases rather than mitigating them. A more sophisticated genre-aware retrieval strategy can be explored to ensure a fairer distribution of metadata across all musical styles.
The choice of a locally hosted LLM, while advantageous in terms of cost, privacy, and control, may also present certain trade-offs. Although our model demonstrates strong performance in metadata generation, it may lack the extensive pretraining and fine-tuning capabilities of larger cloud-based models, which are often trained on more diverse and extensive datasets. A promising direction for future work is to explore hybrid approaches that integrate local inference with selective refinements using advanced cloud-based models, ensuring both efficiency and high-quality metadata generation.
## VI Conclusion
The JamendoMaxCaps dataset represents a significant step forward in enhancing music-language understanding. By addressing metadata sparsity and providing a large-scale, freely accessible dataset, this work enables more effective research in music information retrieval, representation learning, and generation.
## Acknowledgment
This work has received support from SUTD’s Kickstart Initiative under grant number SKI 2021_04_06 and MOE under grant number MOE-T2EP20124-0014. We acknowledge the use of ChatGPT for grammar refinement and paraphrasing.
## References
- [1] A. Agostinelli, T. I. Denk, Z. Borsos, J. Engel, M. Verzetti, A. Caillon, Q. Huang, A. Jansen, A. Roberts, M. Tagliasacchi, M. Sharifi, N. Zeghidour, and C. Frank. Musiclm: Generating music from text, 2023.
- [2] K. Bhandari, A. Roy, K. Wang, G. Puri, S. Colton, and D. Herremans. Text2midi: Generating symbolic music from captions. In Proc. of AAAI, Philadelphia, US, 2025.
- [3] S. Chen, Y. Wu, C. Wang, S. Liu, D. Tompkins, Z. Chen, W. Che, X. Yu, and F. Wei. Beats: Audio pre-training with acoustic tokenizers. In Proc. of the 40th Int. Conf. on Machine Learning (ICML), Honolulu, USA, volume 202 of Proc. of Machine Learning Research (PMLR), 2023.
- [4] Y. Chu, J. Xu, Q. Yang, H. Wei, X. Wei, Z. Guo, Y. Leng, Y. Lv, J. He, J. Lin, et al. Qwen2-audio technical report. arXiv:2407.10759, 2024.
- [5] M. Civit, V. Drai-Zerbib, D. Lizcano, and M. Escalona. Sunocaps: A novel dataset of text-prompt based ai-generated music with emotion annotations. Data in Brief, 55:110743, 2024.
- [6] J. Copet, F. Kreuk, I. Gat, T. Remez, D. Kant, G. Synnaeve, Y. Adi, and A. Défossez. Simple and controllable music generation. In Thirty-seventh Conf. on Neural Information Processing Systems, 2023.
- [7] S. Doh, K. Choi, J. Lee, and J. Nam. Lp-musiccaps: Llm-based pseudo music captioning, 2023.
- [8] Q. Dong, L. Li, D. Dai, C. Zheng, J. Ma, R. Li, H. Xia, J. Xu, Z. Wu, T. Liu, et al. A survey on in-context learning. arXiv:2301.00234, 2022.
- [9] B. Elizalde, S. Deshmukh, M. A. Ismail, and H. Wang. Clap: Learning audio concepts from natural language supervision, 2023.
- [10] Q. Huang, D. S. Park, T. Wang, T. I. Denk, A. Ly, N. Chen, Z. Zhang, Z. Zhang, J. Yu, C. Frank, et al. Noise2music: Text-conditioned music generation with diffusion models. arXiv:2302.03917, 2023.
- [11] J. Kang, S. Poria, and D. Herremans. Video2music: Suitable music generation from videos using an affective multimodal transformer model. Expert Systems with Applications, 249:123640, 2024.
- [12] D. Kim and T. Kim. Missing modality prediction for unpaired multimodal learning via joint embedding of unimodal models. In European Conf. on Computer Vision, pages 171–187. Springer, 2024.
- [13] J. Lee, K. Lee, J. Park, J. Park, and J. Nam. Deep content-user embedding model for music recommendation. arXiv:1807.06786, 2018.
- [14] J. Li, D. Li, S. Savarese, and S. Hoi. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In Proc. of the 40 th Int. Conf. on Machine Learning (ICML), Honolulu, Hawaii, USA, volume 202 of Proc. of Machine Learning Research (PMLR), pages 19730–19742, 2023.
- [15] Y. Li, R. Yuan, G. Zhang, Y. Ma, X. Chen, H. Yin, C. Xiao, C. Lin, A. Ragni, E. Benetos, N. Gyenge, R. B. Dannenberg, R. Liu, W. Chen, G. Xia, Y. Shi, W. Huang, Z. Wang, Y. Guo, and J. Fu. MERT: acoustic music understanding model with large-scale self-supervised training. In The 12th Int. Conf. on Learning Representations, ICLR, Vienna, 2024.
- [16] R. Likert. A technique for the measurement of attitudes. Archives of Psychology, 1932.
- [17] I. Manco, B. Weck, S. Doh, M. Won, Y. Zhang, D. Bogdanov, Y. Wu, K. Chen, et al. The song describer dataset: a corpus of audio captions for music-and-language evaluation. arXiv:2311.10057, 2023.
- [18] X. Mei, C. Meng, H. Liu, Q. Kong, T. Ko, C. Zhao, M. D. Plumbley, Y. Zou, and W. Wang. Wavcaps: A chatgpt-assisted weakly-labelled audio captioning dataset for audio-language multimodal research. IEEE/ACM Trans. Audio Speech Lang. Process., 2024.
- [19] X. Mei, C. Meng, H. Liu, Q. Kong, T. Ko, C. Zhao, M. D. Plumbley, Y. Zou, and W. Wang. WavCaps: A ChatGPT-assisted weakly-labelled audio captioning dataset for audio-language multimodal research. IEEE/ACM Trans. Audio Speech Lang. Process., pages 1–15, 2024.
- [20] J. Melechovsky, Z. Guo, D. Ghosal, N. Majumder, D. Herremans, and S. Poria. Mustango: Toward controllable text-to-music generation. In Proc. of the 2024 Conf. of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 8286–8309, 2024.
- [21] J. Melechovsky, A. Roy, and D. Herremans. Midicaps–a large-scale midi dataset with text captions. In Proc. of ISMIR, 2024.
- [22] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. Bleu: a method for automatic evaluation of machine translation. In Proc. of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318, 2002.
- [23] A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever. Robust speech recognition via large-scale weak supervision. In Proc. of the 40 th Int. Conf. on Machine Learning (ICML), Honolulu, USA, volume 202 of Proc. of Machine Learning Research (PMLR), 2023.
- [24] G. Stoet. Psytoolkit: A software package for programming psychological experiments using linux. Behav. Res. Methods, 42:1096–1104, 2010.
- [25] C. Tang, W. Yu, G. Sun, X. Chen, T. Tan, W. Li, L. Lu, Z. MA, and C. Zhang. SALMONN: Towards generic hearing abilities for large language models. In The 12th Int. Conf. on Learning Representations, 2024.
- [26] Y. Wu, K. Chen, T. Zhang, Y. Hui, T. Berg-Kirkpatrick, and S. Dubnov. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2023.
- [27] T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi. Bertscore: Evaluating text generation with bert. arXiv:1904.09675, 2019.