# Adapting Frechet Audio Distance for Generative Music Evaluation
## Abstract
The growing popularity of generative music models underlines the need for perceptually relevant, objective music quality metrics. The Frechet Audio Distance (FAD) is commonly used for this purpose even though its correlation with perceptual quality is understudied. We show that FAD performance may be hampered by sample size bias, poor choice of audio embeddings, or the use of biased or low-quality reference sets. We propose reducing sample size bias by extrapolating scores towards an infinite sample size. Through comparisons with MusicCaps labels and a listening test we identify audio embeddings and music reference sets that yield FAD scores well-correlated with acoustic and musical quality. Our results suggest that per-song FAD can be useful to identify outlier samples and predict perceptual quality for a range of music sets and generative models. Finally, we release a toolkit that allows adapting FAD for generative music evaluation.
Index Terms — Generative model, Music/Audio generation, Frechet Audio Distance, Evaluation Metrics, Quality Assessment
## 1 Introduction
Advancements in machine learning have accelerated progress in automated music creation. Recently, there has been growing interest in models that directly output music as opposed to a symbolic representation like MIDI [1, 2, 3, 4, 5, 6, 7]. However, evaluating the perceptual quality of music generated by these models remains a challenge.
Various metrics have been proposed for automatic music evaluation. Some metrics prioritize prompt adherence, including the CLAP score [8], MuLan Cycle Consistency [9], and Kullback-Leibler (KL) divergence over audio classifiers. Other metrics emphasize latent proximity toward a “studio-quality” reference set, such as the Inception Score and the increasingly popular Frechet Audio Distance (FAD) [10]. FAD was adapted from the Frechet Inception Distance (FID) used in the image domain. A common approach to calculate FAD is to use the MusicCaps dataset [2] as a reference set and extract audio embeddings using the VGGish model [11], yielding a single FAD score for a set of generated music samples.
Despite being commonly used, prior work suggests that existing objective quality metrics including FAD fail to reliably predict the perceptual quality of generative music [12]. Furthermore, FAD scores published in the literature are often obtained using a variety of sample sizes, reference music sets, and embeddings, all of which may affect FAD performance as a perceptual metric and make comparisons between reported results difficult or impossible.
Here, we examine limitations of FAD and propose improvements for evaluating generative music. Our main contributions are:
- introducing FAD ${}_∞$ (based on FID ${}_∞$ [13]) to reduce sample size bias;
- evaluating per-song FAD to identify outliers in a sample set;
- evaluating two reference sets sourced from public datasets with permissive licensing and a variety of state-of-the-art audio embedding models objectively and subjectively to obtain FAD scores that correlate well with perceptual quality.
We release an FAD toolkit geared towards music evaluation. Code available at github.com/microsoft/fadtk
## 2 Background
Kilgour et al. proposed adapting the Frechet Inception distance (FID) for evaluating music enhancement methods [10]. Similarly to the original FID, the Frechet audio distance (FAD) has since become a commonly used objective metric for evaluating generative models. The FAD is computed by comparing a set of audio samples to a reference set in terms of their respective distributions in an embedding space. Given a multinormal fit to the distribution of audio embedding features with means $μ_r$ and $μ_t$ and covariance matrices $Σ_r$ and $Σ_t$ for the reference and test set, respectively, the FAD is [10]
$$
FAD=\|μ_r-μ_t\|^2+tr≤ft(Σ_r+Σ_t-2√{
Σ_rΣ_t}\right), \tag{1}
$$
where $tr(·)$ is the matrix trace. While FAD is often used as a proxy for perceptual quality [2, 1], the underlying assumption given (1) is that the reference set is of high quality, that the audio embeddings capture features related to quality, that the embedding distribution can be approximated by a multinormal fit, and that the resulting single FAD score for the entire test set is a meaningful metric of model performance. In the following, we test some of these assumptions for several reference sets and audio embedding models and propose improvements to address some limitations of FAD.
## 3 FAD for music evaluation
### 3.1 Reference sets for quality estimation
Since FAD is comparison-based, a lower distance only infers high quality if the reference set is of high quality. In the literature, a commonly used reference set for FAD calculation is MusicCaps which consists of 10 s music-related segments from Youtube videos [2]. However, MusicCaps captions indicate that a large portion of the samples are labeled “low-quality” by the expert raters (see Table 2). Furthermore, upon manual inspection, some MusicCaps samples are not music but music-related tutorials, for example.
We propose two new reference sets based on publicly available music datasets. FMA-Pop consists of 4230 songs from the Free Music Archive (FMA) [14]. We selected the 30 most popular songs in 163 genres as ranked by FMA’s “pop” label. We excluded speech-only genres like ”spoken” and ”poetry”, and non-descriptive genres like ”soundtrack” or ”composed music”. MusCC is a combination of 49 songs from the musdb18 dataset released under a Creative Commons license [15] and all 240 songs published in the MoisesDB dataset under a Non-Commercial Research Community license [16]. MusCC consists of studio-quality, professionally mixed music while FMA-Pop consists of a range of recordings and large genre diversity.
### 3.2 Audio embedding models for quality estimation
FAD is computed on audio embeddings. However, embedding models trained with different architectures, losses, and data may capture different aspects of audio. For the Frechet Inception Distance (FID), studies found that the top ImageNet class highly influences FID scores even though it is not quality-related [17], and that low-probability outliers over the Inception embedding are not necessarily low quality [18]. In the music domain, there appears to be no consensus on which embedding might be preferred for capturing music quality. A common choice is VGGish but other embedding models such as OpenL3 [19], TRILL [20], and MuLan [2] are also used to evaluate various generative models.
To assess the impact of the audio embedding on the perceptual aspects captured by the FAD we evaluate a set of embeddings covering a variety of training objectives: VGGish (audio classifier), CLAP [21] and LAION CLAP [22] (joint audio–text embeddings), MERT [23] (self-supervised audio embedding), CDPAM [24] (audio similarity), and EnCodec [25] and Descript Audio Codec (DAC) [26] (low-rate audio codecs). For CLAP, we use model “sep23”. For LAION CLAP, we use “630k-audioset-best” (L-CLAP aud) and “music_audioset_epoch_15_esc_90.14” (L-CLAP mus). For MERT, we use the last transformer layer (MERT) or the n th layer (MERT L n). For CDPAM we use the models “acoustic” (CDPAM ac) and “content” (CDPAM cont). For EnCodec, EnCodec 48k, and DAC, the embedding is extracted from the final layer of the encoder network prior to quantization. Table 1 summarizes the embeddings.
Table 1: Evaluated embedding models.
| VGGish [11] | 1 (16 kHz) | 128 | .96 s | .96 s |
| --- | --- | --- | --- | --- |
| CLAP [21] | 1 (44.1 kHz) | 1024 | 7 s | 1 s |
| L-CLAP [22] | 1 (48 kHz) | 512 | 10 s | 1 s |
| MERT [23] | 1 (24 kHz) | 768 | 5s | .013 s |
| CDPAM [24] | 1 (22.05 kHz) | 512 | 5s | 1 s |
| EnCodec [25] | 1 (24 kHz) | 128 | - | .013 s |
| EnCodec 48k [25] | 2 (48 kHz) | 128 | 1 s | .99 s |
| DAC [26] | 2 (44.1 kHz) | 1024 | 5 s | .012 s |
### 3.3 Addressing sample size bias
Chong and Forsyth show that the Frechet Inception Distance (FID) decreases as the sample size $N$ increases; the bias coefficient (or rate of change) differs between evaluated models; and that FID converges as $N$ approaches infinity [13]. As seen in Fig. 1, sample size and model biases are also present in FAD, i.e., FAD generally decreases with increasing $N$ but at different rates for different models (e.g., AudioLDM [5] vs. Mubert [27]) and embeddings (e.g., MERT vs. EnCodec). These biases may impact the perceptual relevance of FAD scores. However, prior work introducing generative music models including MusicLM [2], Noise2Music [4], and MusicGen [1] typically does not disclose the sample size or evaluation data.
To correct sample size bias for FAD, we propose a method similar to FID ${}_∞$ [13]. Given a test set with $N$ audio embedding frames, we compute FAD scores for audio frames sampled randomly with replacement from the test set for different sample sizes and fit a linear regression model to the scores to extrapolate the unbiased FAD ${}_∞$ estimate at N = $∞$ . As shown in Fig. 1, the linear regression fits the data reasonably well for various testsets and audio embeddings.
<details>
<summary>x1.png Details</summary>
### Visual Description
## Line Graphs: FAD Performance vs. 1/N for Audio Models
### Overview
The image contains two side-by-side line graphs comparing the performance of audio models across different sample sizes (1/N). The left graph evaluates five models (AudioLDM, FMA-Pop, Mubert, MusicCaps, MusicGen) with FAD scores ranging from 0 to 100. The right graph compares three models (MERT, EnCodec, VGGish) with FAD scores from 0 to 20. All graphs plot FAD against 1/N, with R² values indicating correlation strength.
### Components/Axes
- **Left Graph**:
- **Y-axis**: FAD (0–100)
- **X-axis**: 1/N (0–0.001)
- **Legend**:
- AudioLDM (blue, R²=0.00)
- FMA-Pop (orange, R²=0.95)
- Mubert (yellow, R²=0.30)
- MusicCaps (purple, R²=0.62)
- MusicGen (green, R²=0.78)
- **Right Graph**:
- **Y-axis**: FAD (0–20)
- **X-axis**: 1/N (0–0.001)
- **Legend**:
- MERT (blue, R²=0.99)
- EnCodec (orange, R²=0.62)
- VGGish (yellow, R²=0.18)
### Detailed Analysis
#### Left Graph Trends
1. **AudioLDM (blue)**: Flat line at ~95 FAD, no correlation with 1/N (R²=0.00).
2. **FMA-Pop (orange)**: Linear increase from ~10 to ~20 FAD (R²=0.95).
3. **Mubert (yellow)**: Wavy line (~30–40 FAD), minimal correlation (R²=0.30).
4. **MusicCaps (purple)**: Slight upward trend (~10–15 FAD) (R²=0.62).
5. **MusicGen (green)**: Steady increase from ~5 to ~20 FAD (R²=0.78).
#### Right Graph Trends
1. **MERT (blue)**: Linear rise from ~5 to ~15 FAD (R²=0.99).
2. **EnCodec (orange)**: Volatile line (~10–18 FAD) with peaks (R²=0.62).
3. **VGGish (yellow)**: Flat line (~3–4 FAD), no correlation (R²=0.18).
### Key Observations
- **High R² Values**: MERT (0.99) and FMA-Pop (0.95) show strong linear relationships between FAD and 1/N.
- **Low R² Values**: AudioLDM (0.00) and VGGish (0.18) exhibit near-zero correlation.
- **Performance Variability**: EnCodec’s fluctuating FAD suggests inconsistent performance across sample sizes.
- **Model Efficiency**: MusicGen (R²=0.78) and MERT (R²=0.99) demonstrate the most predictable scaling with sample size.
### Interpretation
The graphs highlight how FAD performance correlates with sample size (1/N) across audio models. High R² values (e.g., MERT, FMA-Pop) indicate models where FAD scales predictably as sample size decreases. Conversely, low R² values (AudioLDM, VGGish) suggest FAD remains stable or insensitive to sample size changes. The right graph’s MERT model stands out with near-perfect linear scaling, implying optimal performance predictability. EnCodec’s volatility (R²=0.62) may reflect architectural complexity or dataset-specific limitations. These trends could inform model selection for tasks requiring sample efficiency or stability.
</details>
Fig. 1: FAD score bias for EnCodec and various testsets (left) and for various embeddings and MusicCaps (right), both with FMA-Pop as reference. For details on Mubert and MusicGen see section 4.4.
## 4 Experimental evaluation
### 4.1 Acoustic Sensitivity Test
Similarly to Kilgour et al. [10], we evaluate the sensitivity of FAD, calculated using different embeddings, to acoustic effects and distortions artificially added to a random sample of 100 songs from FMA-Pop. Five effects are implemented using pedalboard [28]: distortion, low-pass filtering, reverberation, pitch down, and pitch up. Figure 2 illustrates the sensitivity of each embedding to the tested distortions. EnCodec and EnCodec-48k are particularly sensitive to distortion and low-pass filtering. All models seem somewhat sensitive to pitch shifting, while none seem overly sensitive to reverberation. This may indicate that the embeddings are sensitive to potential artifacts introduced by audio effects like pitch shifting, and that EnCodec penalizes acoustic degradation similar to distortion.
<details>
<summary>x2.png Details</summary>
### Visual Description
## Line Graphs: Normalized FAD Performance Across Audio Processing Parameters
### Overview
The image contains six line graphs comparing the normalized Fréchet Audio Distance (FAD) performance of five audio processing algorithms (MERT, L-CLAP aud, EnCodec, EnCodec 48k, VGGish) across different audio distortion parameters. Each graph represents a distinct parameter (Distortion, Low-pass, Reverb, Pitch) with varying scales and trends.
---
### Components/Axes
1. **Top Row (Normalized FAD vs. Parameter):**
- **Left Graph (Distortion [dB]):**
- X-axis: Distortion [dB] (range: -10 to 20)
- Y-axis: Normalized FAD (range: 0 to 200)
- **Middle Graph (Low-pass [Hz]):**
- X-axis: Low-pass [Hz] (range: 0 to 4000)
- Y-axis: Normalized FAD (range: 0 to 30)
- **Right Graph (Reverb [damping]):**
- X-axis: Reverb [damping] (range: 0 to 1)
- Y-axis: Normalized FAD (range: 0 to 2.5)
2. **Bottom Row (Normalized FAD vs. Pitch):**
- **Left Graph (Pitch [semitones]):**
- X-axis: Pitch [semitones] (range: -10 to 0)
- Y-axis: Normalized FAD (range: 0 to 15)
- **Right Graph (Pitch [semitones]):**
- X-axis: Pitch [semitones] (range: 0 to 10)
- Y-axis: Normalized FAD (range: 0 to 4)
3. **Legend (Right Side):**
- **Colors and Labels:**
- Blue: MERT
- Red: L-CLAP aud
- Orange: EnCodec
- Purple: EnCodec 48k
- Green: VGGish
---
### Detailed Analysis
#### Top Row
1. **Distortion [dB]:**
- **Trends:**
- MERT (blue): Sharp upward slope (5 → 15 FAD).
- L-CLAP aud (red): Gradual increase (2 → 10 FAD).
- EnCodec (orange): Steep rise (1 → 12 FAD).
- EnCodec 48k (purple): Moderate increase (3 → 18 FAD).
- VGGish (green): Flat line (~1 FAD).
- **Key Data Points:**
- At 20 dB distortion, MERT reaches ~15 FAD, while VGGish remains near 1.
2. **Low-pass [Hz]:**
- **Trends:**
- MERT (blue): Sharp decline (20 → 5 FAD).
- L-CLAP aud (red): Slight decrease (10 → 6 FAD).
- EnCodec (orange): Moderate drop (15 → 8 FAD).
- EnCodec 48k (purple): Slight increase (2 → 3.5 FAD).
- VGGish (green): Flat line (~1 FAD).
- **Key Data Points:**
- At 4000 Hz low-pass, EnCodec 48k shows the highest FAD (~3.5).
3. **Reverb [damping]:**
- **Trends:**
- MERT (blue): Slight increase (1.8 → 2.2 FAD).
- L-CLAP aud (red): Stable (~1.5 FAD).
- EnCodec (orange): Slight decrease (1.2 → 1.0 FAD).
- EnCodec 48k (purple): Flat (~2.0 FAD).
- VGGish (green): Stable (~0.5 FAD).
- **Key Data Points:**
- At 1.0 damping, MERT reaches ~2.2 FAD, while VGGish remains at ~0.5.
#### Bottom Row
1. **Pitch [semitones] (Negative Range):**
- **Trends:**
- MERT (blue): Sharp decline (4 → 1 FAD).
- L-CLAP aud (red): Gradual drop (3 → 1.5 FAD).
- EnCodec (orange): Steep decrease (5 → 2 FAD).
- EnCodec 48k (purple): Moderate drop (4 → 2.5 FAD).
- VGGish (green): Flat (~1 FAD).
- **Key Data Points:**
- At -10 semitones, MERT reaches ~1 FAD, while VGGish stays near 1.
2. **Pitch [semitones] (Positive Range):**
- **Trends:**
- MERT (blue): Flat line (~3 FAD).
- L-CLAP aud (red): Slight increase (1.5 → 2.5 FAD).
- EnCodec (orange): Steep rise (2 → 3.5 FAD).
- EnCodec 48k (purple): Moderate increase (2.5 → 3.0 FAD).
- VGGish (green): Flat (~1 FAD).
- **Key Data Points:**
- At 10 semitones, EnCodec reaches ~3.5 FAD, while VGGish remains at ~1.
---
### Key Observations
1. **VGGish Consistency:** VGGish (green) maintains the lowest and most stable FAD across all parameters, suggesting robustness to distortions.
2. **EnCodec 48k Variability:** EnCodec 48k (purple) shows mixed performance, with higher FAD in some cases (e.g., low-pass, reverb) but lower in others (e.g., pitch).
3. **MERT Sensitivity:** MERT (blue) exhibits significant FAD increases with distortion and low-pass parameters but performs well in pitch adjustments.
4. **L-CLAP aud Stability:** L-CLAP aud (red) shows moderate FAD across most parameters but spikes in the positive pitch range.
---
### Interpretation
The data suggests that **VGGish** is the most robust algorithm for handling audio distortions, maintaining low FAD values across all tested parameters. **EnCodec 48k** and **MERT** show parameter-specific weaknesses, with EnCodec 48k struggling in low-pass and reverb scenarios and MERT being sensitive to distortion. **L-CLAP aud** performs consistently but lags behind VGGish in critical cases. These trends highlight the importance of algorithm selection based on the target audio processing task (e.g., distortion vs. pitch correction).
</details>
Fig. 2: Impact of audio effects on FAD scores (normalized by FAD for unprocessed samples to show relative change).
### 4.2 Predicting MusicCaps quality with per-song FAD
We evaluate the FAD scores’ effectiveness for capturing perceptual aspects related to acoustic quality (AQ), i.e., audio recording quality in terms of noise and artifacts, and musical quality (MQ), i.e., quality of the musical composition and performance.
We analyse MusicCaps captions using GPT-4 via the Azure OpenAI service to obtain subjective AQ and MQ labels “high”, “medium”, “low”, or “not mentioned” for each song. As an example, the caption “The low quality recording features a ballad song that contains sustained strings, mellow piano melody and soft female vocal singing over it. It sounds sad and soulful, like something you would hear at Sunday services.” is assigned the labels “low” for AQ and “high” for MQ. The AQ and MQ labels extracted from MusicCaps captions are summarised in Table 2. To balance the labels we aggregate AQ labels into the classes “low” and “not low”, and MQ labels into the classes “high” and “not high”. GPT-4 MusicCaps labels available at github.com/microsoft/fadtk
Table 2: Acoustic (AQ) and musical quality (MQ) labels extracted from MusicCaps captions using GPT4.
| Acoustic quality Musical quality | 1% 42% | 2% 9% | 39% 9% | 58% 41% |
| --- | --- | --- | --- | --- |
<details>
<summary>x3.png Details</summary>
### Visual Description
## Chart: F1-Score Comparison Across Audio Classification Models
### Overview
The image presents a dual-axis bar chart comparing F1-scores for audio classification models across two metrics: "F1-score AQ" (top subplot) and "F1-score MQ" (bottom subplot). Three datasets are compared: MusCC (light gray), FMA-Pop (dark gray), and MusicCaps (black). The x-axis lists 11 audio classification models or categories, while the y-axis ranges from 0.4 to 0.9 for F1-score AQ and 0.5 to 0.9 for F1-score MQ.
### Components/Axes
- **X-axis (Models/Categories)**:
L-CLAP aud, L-CLAP mus, CLAP, EnCodec, DAC, MERT L4, CDPAM ac, CDPAM cont, EnCodec 48k, VGGish, random
- **Y-axis (F1-score)**:
Top subplot: 0.4–0.9 (F1-score AQ)
Bottom subplot: 0.5–0.9 (F1-score MQ)
- **Legend**:
- MusCC: Light gray squares
- FMA-Pop: Dark gray diamonds
- MusicCaps: Black circles
### Detailed Analysis
#### Top Subplot (F1-score AQ)
- **Trend**: All datasets show moderate performance, with scores clustered between 0.5 and 0.7.
- **Key Data Points**:
- **L-CLAP aud**: MusCC (~0.62), FMA-Pop (~0.65), MusicCaps (~0.68)
- **EnCodec**: MusCC (~0.60), FMA-Pop (~0.63), MusicCaps (~0.66)
- **VGGish**: MusCC (~0.55), FMA-Pop (~0.58), MusicCaps (~0.61)
- **Random**: MusCC (~0.45), FMA-Pop (~0.48), MusicCaps (~0.50)
#### Bottom Subplot (F1-score MQ)
- **Trend**: Scores are generally higher (0.6–0.8), with MusicCaps consistently outperforming others.
- **Key Data Points**:
- **CLAP**: MusCC (~0.72), FMA-Pop (~0.75), MusicCaps (~0.78)
- **DAC**: MusCC (~0.70), FMA-Pop (~0.73), MusicCaps (~0.76)
- **EnCodec 48k**: MusCC (~0.68), FMA-Pop (~0.71), MusicCaps (~0.74)
- **Random**: MusCC (~0.55), FMA-Pop (~0.58), MusicCaps (~0.60)
### Key Observations
1. **MusicCaps Dominance**: MusicCaps achieves the highest F1-scores in both subplots, particularly in F1-score MQ (e.g., 0.78 for CLAP).
2. **VGGish Underperformance**: VGGish scores the lowest across most categories, with F1-score AQ dropping to ~0.55.
3. **Random Baseline**: The "random" category serves as a lower bound, with scores ~0.45–0.60.
4. **Subplot Divergence**: F1-score MQ values are consistently higher than F1-score AQ, suggesting MQ metric is less stringent.
### Interpretation
The data suggests **MusicCaps** is the most robust audio classification model across all categories, likely due to its specialized training on MusicCaps dataset. **VGGish** struggles with lower scores, indicating potential limitations in feature extraction for these tasks. The **F1-score MQ** metric appears more lenient, as its range starts at 0.5 compared to 0.4 for F1-score AQ.
Notably, **EnCodec 48k** and **DAC** show strong performance in F1-score MQ (~0.71–0.76), suggesting they handle MQ tasks effectively despite moderate AQ scores. The "random" category’s performance highlights the baseline difficulty of the classification tasks.
The spatial grounding of the legend (center-right) and consistent symbol-color alignment confirm data integrity. No anomalies are observed beyond expected performance variations.
</details>
Fig. 3: Precision (diamond), recall (square), and F1 score (circle) of FAD predicting MusicCaps acoustic (AQ, top) and musical quality (MQ, bottom) with FMA-Pop, MusCC, or MusicCaps as reference. The F1 score is the harmonic mean of precision and recall.
For each audio embedding, we calculate an individual FAD score for every song in the MusicCaps dataset using FMA-Pop, MusCC, and MusicCaps itself as reference and select the top and bottom 5% (i.e., 276 songs) according to their FAD score. To test whether these FAD scores are perceptually meaningful, we assign the AQ label “low” to the songs with the highest FAD scores and the MQ label “high” to the songs with the lowest FAD scores and compare these FAD-predicted labels with the AQ and MQ labels extracted from the corresponding MusicCaps captions using GPT4.
Figure 3 illustrates the performance of per-song FAD scores for predicting MusicCaps AQ and MQ labels in terms of precision, recall, and F1 score. CLAP and L-CLAP outperform other embeddings overall, perhaps indicating that the joint audio–text embeddings capture both acoustic and musical characteristics. However, they see a drop in performance for AQ prediction when using MusicCaps as a reference set, possibly due to the presence of low-quality samples in MusicCaps (cf. Table 2). EnCodec performs well for AQ prediction, in line with its high sensitivity to acoustic distortions (cf. Section 4.1). Among all MERT transformer layers, we found layer 4 to produce the best results, performing similarly to CLAP and L-CLAP for MQ prediction. This is in line with results reported by MERT authors indicating that earlier transformer layers may show perform better for certain music information retrieval tasks [23]. VGGish predicts MQ labels only slightly better than chance, perhaps due to a lack of music-specific labels or data in training.
### 4.3 Identifying outliers with per-song FAD
To further gauge how the choice of audio embedding and reference set might impact which acoustic and musical characteristics are captured by FAD we check the MusicCaps musical excerpts that receive the five highest and five lowest per-song FAD for various combinations of audio embedding and reference set. Examples available at fadtk.hydev.org For MERT L4, MusicCaps samples that consist of synthetic tones with little or no temporal or spectral variation consistently receive the highest FAD scores, irrespective of the reference set used, indicating per-song FAD with MERT may be useful to identify samples of low musical information. Among the excerpts with the highest FAD scores using L-CLAP aud are non-musical field recordings, a speech-only sample, and monophonic synthesizer sounds. The excerpts with the lowest FAD scores for L-CLAP aud consist of studio-quality music, but the genre diversity seems to depend on the reference set. When using MusCC, all five excerpts with the lowest FAD score fall within the Western music genres Pop and Rock (consistent with the dominant genres in MusCC), whereas using FMA-Pop the five excerpts span a variety of genres, including Jazz and Latin music. This suggests that genre-imbalance in the reference set may bias FAD scores. For EnCodec, the highest FAD scores are assigned to samples with heavy distortion or clipping. This is again in line with results from the acoustic sensitivity test (see Section 4.1). In the case of VGGish embeddings with MusicCaps as a reference, i.e., a common choice for FAD calculation in the generative music literature, four out of five samples with the highest FAD scores consist of bagpipe music. Interestingly, Kong et al. report that for AudioSet classification (i.e., the same training objective as VGGish) the class “bagpipes” achieves the highest average precision out of all 527 classes [29]. It is possible that particularly recognizable classes may produce unique embedding activations and thus result in high FAD scores. These anecdotal findings suggest that per-song FAD scores may be a useful tool to find outliers (e.g., heavily distorted, noisy, or non-musical samples) or to check whether the chosen audio embedding and reference set may introduce a bias with respect to musical genre or instruments.
### 4.4 Subjective evaluation
<details>
<summary>x4.png Details</summary>
### Visual Description
## Box Plot: Mean Opinion Score (MOS) Comparison Across Audio Quality (AQ) and Model Quality (MQ)
### Overview
The image contains four box plots comparing Mean Opinion Scores (MOS) across different audio quality categories (AQ) and model quality categories (MQ). Each panel uses a vertical y-axis labeled "MOS" (1–5) and a horizontal x-axis with categorical labels. The plots include medians (dots), interquartile ranges (boxes), whiskers (range), and outliers (circles).
### Components/Axes
- **Y-Axis**: "MOS" (Mean Opinion Score), scaled from 1 to 5 in increments of 1.
- **X-Axes**:
- **AQ Panel (Left)**: Categories labeled "Low," "N/A," "High," "Medium," "Low," "N/A."
- **MQ Panel (Right)**: Categories labeled "FMA-Pop," "MusicCaps," "MusicGen," "MusicLM," "Mubert."
- **Legend**: Not explicitly labeled, but visual markers differentiate medians (solid dots) and outliers (open circles).
### Detailed Analysis
#### AQ Panel (Left)
1. **Low**: Median ≈ 3.0, range 2.0–4.0, outliers at 2.5 and 4.5.
2. **N/A**: Median ≈ 3.5, range 2.5–4.5, outliers at 2.0 and 5.0.
3. **High**: Median ≈ 4.0, range 3.0–4.5, outliers at 2.0 and 5.0.
4. **Medium**: Median ≈ 3.0, range 2.0–4.0, outliers at 1.5 and 4.5.
5. **Low (repeated)**: Median ≈ 3.0, range 2.0–4.0, outliers at 2.5 and 4.5.
6. **N/A (repeated)**: Median ≈ 3.5, range 2.5–4.5, outliers at 2.0 and 5.0.
#### MQ Panel (Right)
1. **FMA-Pop**: Median ≈ 3.5, range 2.5–4.5, outliers at 2.0 and 5.0.
2. **MusicCaps**: Median ≈ 3.0, range 2.0–4.0, outliers at 1.5 and 4.5.
3. **MusicGen**: Median ≈ 3.5, range 2.5–4.5, outliers at 2.0 and 5.0.
4. **MusicLM**: Median ≈ 3.0, range 2.0–4.0, outliers at 1.5 and 4.5.
5. **Mubert**: Median ≈ 3.5, range 2.5–4.5, outliers at 2.0 and 5.0.
### Key Observations
- **AQ Trends**: Higher AQ categories (e.g., "High") generally show higher medians (≈4.0) compared to "Low" or "Medium" (≈3.0). However, overlap in ranges suggests variability.
- **MQ Trends**: Models like "FMA-Pop" and "Mubert" have higher medians (≈3.5) than "MusicCaps" or "MusicLM" (≈3.0). Outliers are frequent across all categories.
- **Outliers**: Open circles indicate extreme values, often at the lower (e.g., 1.5–2.0) or upper (4.5–5.0) bounds of the MOS scale.
### Interpretation
The data suggests that **audio quality (AQ)** and **model quality (MQ)** significantly influence MOS. Higher AQ categories correlate with higher median scores, while certain MQ models (e.g., "FMA-Pop," "Mubert") outperform others. However, the presence of outliers and overlapping ranges indicates inconsistency in human evaluations. The repeated "Low" and "N/A" labels in the AQ panel may imply redundant or mislabeled categories, warranting further clarification.
### Spatial Grounding
- **Legend**: Not explicitly present; medians (solid dots) and outliers (open circles) are visually distinct.
- **Panel Layout**: AQ and MQ panels are side-by-side, with AQ on the left and MQ on the right.
- **Axis Alignment**: All panels share the same y-axis (MOS), enabling direct comparison of medians and ranges.
### Content Details
- **AQ Categories**:
- "Low": Median = 3.0, range = 2.0–4.0.
- "High": Median = 4.0, range = 3.0–4.5.
- **MQ Models**:
- "MusicGen": Median = 3.5, range = 2.5–4.5.
- "MusicCaps": Median = 3.0, range = 2.0–4.0.
### Notable Anomalies
- The "N/A" category in AQ has a higher median (3.5) than "Low" (3.0), which may reflect subjective interpretations of "N/A" as neutral or mid-range.
- Outliers in "MusicCaps" and "MusicLM" suggest extreme evaluations, possibly due to model-specific artifacts or user biases.
### Peircean Investigative Insights
The repeated "Low" and "N/A" labels in the AQ panel raise questions about data categorization. If "N/A" represents missing or undefined data, its inclusion in the analysis may skew results. Additionally, the clustering of outliers at the extremes (1.5–2.0 and 4.5–5.0) suggests polarized opinions, potentially driven by technical limitations or subjective preferences.
</details>
Fig. 4: Mean opinion score (MOS) of acoustic quality (AQ) and musical quality (MQ) for MusicCaps labels extracted by GPT4 (left two panels) and for all datasets in the listening test (right two panels). Note: The left two panels consist of only the MusicCaps samples within the listening test subset and not the entire MusicCaps set.
To evaluate the performance of per-song FAD scores calculated using various audio embeddings and reference sets for predicting subjective quality of generative music we conducted a listening test. The test dataset consisted of 300 10 s excerpts from five sources: 100 random songs from FMA-Pop, 50 random songs from MusicCaps [2], and 50 songs each generated by Mubert [27], MusicLM [2], and MusicGen (Large) [1]. The prompts for the musical excerpts generated using Mubert, MusicLM, and MusicGen were derived from the 50 MusicCaps captions via GPT-4 with a one-shot example. GPT-4 prompt and results available at github.com/microsoft/fadtk During the test, participants were asked to rate each excerpt in terms of acoustic quality (AQ) and musical quality (MQ) on a five-point Likert scale. The order of excerpts was fully randomized. We recruited 10 participants within Microsoft. Each participant completed the test in approximately one hour using a pair of AKG K271 MkII headphones. All participants received a $25 gift card. The study design was approved by Microsoft’s Ethics Review Board.
Figure 4 shows the Mean Opinion Score (MOS) distribution for all testsets in the listening test. As can be seen , a “low” AQ rating assigned by GPT-4 indeed corresponds to a lower MOS compared to “N/A’ (i.e., no label assigned). Similarly, “high”, “medium”, and “low” MQ labels assigned by GPT-4 correlate well with MOS results. Among the five testsets, Mubert achieves the highest median MOS for AQ, followed by FMA-Pop. For MQ, the differences between the testsets are less pronounced. Note that listeners were not asked to rate prompt adherence.
Figure 5 summarizes the Pearson correlation coefficients (PCCs) between per-song FAD and MOS. A PCC is calculated separately for each of the five testsets to measure per-song FAD performance for predicting MOS within each testset, and to account for potential model or dataset bias (cf. Section 3.3). Similarly to the results for MusicCaps label prediction, CLAP and L-CLAP outperform other embeddings for AQ and MQ MOS prediction. DAC and EnCodec perform similarly. In line with MusicCaps results, MERT L4 performs poorly for AQ MOS prediction and fares better for MQ. Finally, VGGish does not correlate well with either AQ or MQ MOS. When comparing the three reference sets used for FAD calculation, MusCC seems to boost MQ MOS prediction compared to MusicCaps especially for CLAP and L-CLAP models, whereas for AQ prediction the differences are less clear.
<details>
<summary>x5.png Details</summary>
### Visual Description
## Box Plot Chart: Performance Comparison of Music Models Across Datasets
### Overview
The image displays two vertically stacked box plots comparing the performance of three music models (MusCC, FMA-Pop, MusicCaps) across 12 datasets. The top subplot measures "PCC AQ" (Pearson Correlation Coefficient - Audio Quality), while the bottom subplot measures "PCC MQ" (Pearson Correlation Coefficient - Music Quality). Each box plot shows the distribution of scores for each model-dataset pair, with medians, quartiles, and outliers marked.
---
### Components/Axes
- **X-axis**: Dataset names (e.g., "L-CLAP aud", "mus", "CLAP", "DAC", "EnCodec", "MERT L4", "CDPAM ac", "EnCodec 48k", "CDPAM cont", "VGGish").
- **Y-axis (Top Subplot)**: "PCC AQ" (range: -0.8 to 0.5).
- **Y-axis (Bottom Subplot)**: "PCC MQ" (range: -0.8 to 0.2).
- **Legend**: Located in the top-left corner, with three color-coded models:
- **MusCC**: Light gray (■)
- **FMA-Pop**: Dark gray (■)
- **MusicCaps**: Black (■)
---
### Detailed Analysis
#### Top Subplot (PCC AQ)
- **Median Values**:
- **MusicCaps** consistently shows the highest medians (e.g., ~0.3 in "MERT L4", ~0.2 in "CDPAM ac").
- **FMA-Pop** has the lowest medians (e.g., ~-0.2 in "MERT L4", ~-0.1 in "CDPAM ac").
- **MusCC** falls between the two, with medians ranging from ~-0.1 to ~0.1.
- **Outliers**:
- **MusicCaps** has outliers in "EnCodec" (~0.4) and "CDPAM 48k" (~0.3).
- **FMA-Pop** has outliers in "EnCodec" (~-0.3) and "CDPAM 48k" (~-0.2).
- **Trends**:
- MusicCaps generally outperforms other models, with a slight upward trend in median values across datasets.
- FMA-Pop shows the most variability, with some datasets (e.g., "EnCodec") having lower medians.
#### Bottom Subplot (PCC MQ)
- **Median Values**:
- **MusicCaps** again leads, with medians around ~-0.1 to ~0.1.
- **FMA-Pop** has the lowest medians (e.g., ~-0.3 in "MERT L4", ~-0.2 in "CDPAM ac").
- **MusCC** shows moderate performance, with medians between ~-0.2 and ~0.0.
- **Outliers**:
- **MusicCaps** has outliers in "EnCodec" (~-0.4) and "CDPAM 48k" (~-0.3).
- **FMA-Pop** has outliers in "EnCodec" (~-0.5) and "CDPAM 48k" (~-0.4).
- **Trends**:
- MusicCaps maintains higher performance than FMA-Pop, but both models show negative medians in some datasets.
---
### Key Observations
1. **Model Performance**:
- **MusicCaps** consistently outperforms **FMA-Pop** and **MusCC** in both PCC AQ and PCC MQ across most datasets.
- **FMA-Pop** exhibits the most variability, with lower medians and more outliers in certain datasets (e.g., "EnCodec", "CDPAM 48k").
2. **Dataset Variability**:
- **EnCodec** and **CDPAM 48k** show the most significant deviations, with outliers in both subplots.
- **MERT L4** and **CDPAM ac** demonstrate the highest performance for MusicCaps in PCC AQ.
3. **Negative Values**:
- Both subplots include negative values, indicating some datasets where models underperformed relative to the baseline.
---
### Interpretation
The data suggests that **MusicCaps** is the most effective model for audio and music quality metrics across the tested datasets, with **FMA-Pop** showing the weakest performance. The presence of outliers in specific datasets (e.g., "EnCodec", "CDPAM 48k") highlights potential inconsistencies in model behavior under certain conditions. The negative values in some datasets may indicate limitations in model generalizability or dataset-specific challenges. This comparison underscores the importance of model selection based on dataset characteristics and evaluation criteria.
</details>
Fig. 5: Pearson correlation coefficient (PCC) between FAD and listening test MOS for all tested embeddings and reference datasets, for acoustic (AQ, top) and musical quality (MQ, bottom). Each box summarizes the five PCCs of the five testsets; lower PCC is better. Each box on the boxplot shows five PCCs associated with the five testsets.
## 5 Conclusion
Automated evaluation of generative music models remains a challenging problem. Despite its limitations, the Frechet Audio Distance (FAD) is a promising metric for rating the quality of generative music. We evaluated several state-of-the-art audio embeddings for FAD calculation as well as two new reference sets derived from public datasets. Our experimental results are in line with prior work suggesting that sample size bias in FAD calculation exists but can be estimated and mitigated [13], and that embeddings trained on a classification objective such as VGGish may not be optimal for FAD calculation [18, 17]. Evaluations on subjective labels extracted from MusicCaps captions as well as a listening test indicate that a proper choice of reference dataset and audio embedding results in FAD scores that meaningfully correlate with subjective ratings of generative music in terms of acoustic and musical quality. Furthermore, the results suggest that FAD scores calculated for individual samples rather than an entire testset at once may prove useful for identifying outliers, including samples with severe artifacts, atypical musical characteristics, or non-music signals. We hope these results along with the resources published with our FAD toolkit support future work on generative music.
## References
- [1] J. Copet, F. Kreuk, I. Gat, T. Remez, D. Kant, G. Synnaeve, Y. Adi, and A. Défossez, “Simple and Controllable Music Generation,” June 2023, arXiv:2306.05284 [cs, eess].
- [2] A. Agostinelli, T. I. Denk, Z. Borsos, J. Engel, M. Verzetti, A. Caillon, Q. Huang, A. Jansen, A. Roberts, M. Tagliasacchi, M. Sharifi, N. Zeghidour, and C. Frank, “MusicLM: Generating Music From Text,” Jan. 2023, arXiv:2301.11325 [cs, eess].
- [3] D. Yang, J. Yu, H. Wang, W. Wang, C. Weng, Y. Zou, and D. Yu, “Diffsound: Discrete Diffusion Model for Text-to-sound Generation,” Apr. 2023, arXiv:2207.09983 [cs, eess].
- [4] Q. Huang, D. S. Park, T. Wang, T. I. Denk, A. Ly, N. Chen, Z. Zhang, Z. Zhang, J. Yu, C. Frank, J. Engel, Q. V. Le, W. Chan, Z. Chen, and W. Han, “Noise2Music: Text-conditioned Music Generation with Diffusion Models,” Mar. 2023, arXiv:2302.03917 [cs, eess].
- [5] H. Liu, Z. Chen, Y. Yuan, X. Mei, X. Liu, D. Mandic, W. Wang, and M. D. Plumbley, “AudioLDM: Text-to-Audio Generation with Latent Diffusion Models,” Feb. 2023, arXiv:2301.12503 [cs, eess].
- [6] G. Mariani, I. Tallini, E. Postolache, M. Mancusi, L. Cosmo, and E. Rodolà, “Multi-Source Diffusion Models for Simultaneous Music Generation and Separation,” Feb. 2023, arXiv:2302.02257 [cs, eess].
- [7] K. Goel, A. Gu, C. Donahue, and C. Ré, “It’s Raw! Audio Generation with State-Space Models,” Feb. 2022, arXiv:2202.09729 [cs, eess].
- [8] B. Elizalde, S. Deshmukh, M. A. Ismail, and H. Wang, “CLAP: Learning Audio Concepts From Natural Language Supervision,” June 2022, arXiv:2206.04769 [cs, eess].
- [9] Q. Huang, A. Jansen, J. Lee, R. Ganti, J. Y. Li, and D. P. W. Ellis, “MuLan: A Joint Embedding of Music Audio and Natural Language,” Aug. 2022, arXiv:2208.12415 [cs, eess, stat].
- [10] K. Kilgour, M. Zuluaga, D. Roblek, and M. Sharifi, “Fr\’echet Audio Distance: A Metric for Evaluating Music Enhancement Algorithms,” Jan. 2019, arXiv:1812.08466 [cs, eess].
- [11] S. Hershey, S. Chaudhuri, D. P. W. Ellis, J. F. Gemmeke, A. Jansen, R. C. Moore, M. Plakal, D. Platt, R. A. Saurous, B. Seybold, M. Slaney, R. J. Weiss, and K. Wilson, “CNN Architectures for Large-Scale Audio Classification,” Jan. 2017, arXiv:1609.09430 [cs, stat].
- [12] A. Vinay and A. Lerch, “Evaluating generative audio systems and their metrics,” Aug. 2022, arXiv:2209.00130 [cs, eess].
- [13] M. J. Chong and D. Forsyth, “Effectively Unbiased FID and Inception Score and where to find them,” June 2020, arXiv:1911.07023 [cs].
- [14] M. Defferrard, K. Benzi, P. Vandergheynst, and X. Bresson, “FMA: A Dataset For Music Analysis,” Sept. 2017, arXiv:1612.01840 [cs].
- [15] Z. Rafii, A. Liutkus, F.-R. Stöter, S. I. Mimilakis, and R. Bittner, “The MUSDB18 corpus for music separation,” Dec. 2017.
- [16] I. Pereira, F. Araújo, F. Korzeniowski, and R. Vogl, “Moisesdb: A dataset for source separation beyond 4-stems,” 2023.
- [17] T. Kynkäänniemi, T. Karras, M. Aittala, T. Aila, and J. Lehtinen, “The Role of ImageNet Classes in Fr\’echet Inception Distance,” Feb. 2023, arXiv:2203.06026 [cs, stat].
- [18] E. Betzalel, C. Penso, A. Navon, and E. Fetaya, “A Study on the Evaluation of Generative Models,” June 2022, arXiv:2206.10935 [cs].
- [19] A. L. Cramer, H.-H. Wu, J. Salamon, and J. P. Bello, “OpenL3 - Look, Listen, and Learn More: Design Choices for Deep Audio Embeddings,” in ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), May 2019, pp. 3852–3856, ISSN: 2379-190X.
- [20] J. Shor, A. Jansen, R. Maor, O. Lang, O. Tuval, F. d. C. Quitry, M. Tagliasacchi, I. Shavitt, D. Emanuel, and Y. Haviv, “TRILL - Towards Learning a Universal Non-Semantic Representation of Speech,” in Interspeech 2020, Oct. 2020, pp. 140–144, arXiv:2002.12764 [cs, eess, stat].
- [21] B. Elizalde, S. Deshmukh, and H. Wang, “Natural language supervision for general-purpose audio representations,” submitted to IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2024.
- [22] Y. Wu*, K. Chen*, T. Zhang*, Y. Hui*, T. Berg-Kirkpatrick, and S. Dubnov, “Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,” in IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP, 2023.
- [23] Y. Li, R. Yuan, G. Zhang, Y. Ma, X. Chen, H. Yin, C. Lin, A. Ragni, E. Benetos, N. Gyenge, R. Dannenberg, R. Liu, W. Chen, G. Xia, Y. Shi, W. Huang, Y. Guo, and J. Fu, “MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Training,” June 2023, arXiv:2306.00107 [cs, eess].
- [24] P. Manocha, Z. Jin, R. Zhang, and A. Finkelstein, “CDPAM: Contrastive learning for perceptual audio similarity,” Feb. 2021, arXiv:2102.05109 [cs, eess].
- [25] A. Défossez, J. Copet, G. Synnaeve, and Y. Adi, “High Fidelity Neural Audio Compression,” Oct. 2022, arXiv:2210.13438 [cs, eess, stat].
- [26] R. Kumar, P. Seetharaman, A. Luebs, I. Kumar, and K. Kumar, “High-Fidelity Audio Compression with Improved RVQGAN,” June 2023, arXiv:2306.06546 [cs, eess].
- [27] Mubert-Inc, “Mubert,” 2023, Available at: https://mubert.com/.
- [28] P. Sobot, “Pedalboard,” July 2021.
- [29] Q. Kong, Y. Cao, T. Iqbal, Y. Wang, W. Wang, and M. D. Plumbley, “Panns: Large-scale pretrained audio neural networks for audio pattern recognition,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 28, pp. 2880–2894, 2020.