# MuQ-Eval: An Open-Source Per-Sample Quality Metric for AI Music Generation Evaluation
**Authors**: Di Zhuand Zixuan Li
## Abstract
Distributional metrics such as Fréchet Audio Distance cannot score individual music clips and correlate poorly with human judgments, while the only per-sample learned metric achieving high human correlation is closed-source. We introduce MuQ-Eval, an open-source per-sample quality metric for AI-generated music built by training lightweight prediction heads on frozen MuQ-310M features using MusicEval, a dataset of generated clips from 31 text-to-music systems with expert quality ratings. Our simplest model, frozen features with attention pooling and a two-layer MLP, achieves system-level SRCC = 0.957 and utterance-level SRCC = 0.838 with human mean opinion scores. A systematic ablation over training objectives and adaptation strategies shows that no addition meaningfully improves the frozen baseline, indicating that frozen MuQ representations already capture quality-relevant information. Encoder choice is the dominant design factor, outweighing all architectural and training decisions. LoRA-adapted models trained on as few as 150 clips already achieve usable correlation, enabling personalized quality evaluators from individual listener annotations. A controlled degradation analysis reveals selective sensitivity to signal-level artifacts but insensitivity to musical-structural distortions. Our metric, MuQ-Eval, is fully open-source, outperforms existing open per-sample metrics, and runs in real time on a single consumer GPU. Code, model weights, and evaluation scripts are available at https://github.com/dgtql/MuQ-Eval.
## I Introduction
Text-to-music (TTM) generation has advanced rapidly. Systems such as MusicGen [1], MusicLM [2], and Stable Audio [3] can now produce multi-instrument arrangements from short text prompts, and the pace of new model releases continues to accelerate. Yet this progress has exposed a critical bottleneck: the field lacks reliable, automatic methods for assessing the perceptual quality of generated music. Without such methods, researchers cannot rigorously compare systems, practitioners cannot filter low-quality outputs at scale, and reinforcement-from-feedback training loops lack a trustworthy reward signal. Quality assessment has consequently emerged as one of the most pressing open problems in music generation research.
The difficulty is fundamental. Music quality is a multi-dimensional percept encompassing timbral naturalness, harmonic coherence, rhythmic stability, structural plausibility, and production polish, attributes that are difficult to define formally and expensive to annotate by human listeners. Unlike image generation, where perceptual similarity to a reference can be measured pixel-wise, or speech synthesis, where intelligibility and naturalness are well-operationalized, music quality has no widely accepted ground-truth signal. Human listening studies remain the gold standard, but they are slow, costly, and poorly reproducible across labs, making them impractical as a routine evaluation tool during model development.
Existing automatic metrics fail to fill this gap. Distributional metrics such as Fréchet Audio Distance (FAD) [4] compute a single scalar between sets of generated and reference audio, making them unable to score individual clips. Their correlation with human preferences is weak and embedding-dependent: VGGish-based FAD achieves only $τ=0.14$ with human rankings [5], and even the best distributional variant, MAD with MERT embeddings, reaches $τ=0.62$ [5], far below the per-sample correlations routinely achieved in speech quality prediction. Text-audio alignment scores such as CLAP [6, 7] measure semantic relevance rather than perceptual quality: a clip can receive a high CLAP score while sounding severely degraded. The per-sample learned metrics that do exist are either weakly correlated with music quality (Audiobox Aesthetics: $r=0.200$ [8]) or closed-source (DORA-MOS: SRCC $=0.988$ [9]). In short, the music generation community currently has no open, per-sample quality metric that correlates well with human perception.
This stands in sharp contrast to adjacent perceptual domains where the combination of deep features from domain-specific encoders with human quality annotations has produced highly effective learned metrics. LPIPS [10] achieves 68.9% two-alternative forced choice agreement with human judgments on image quality. In speech, NISQA [11] reaches PCC $=0.80$ – $0.95$ and DNSMOS [12] achieves PCC $=0.94$ – $0.98$ for mean opinion score prediction. The AudioMOS 2025 Challenge [9] demonstrated that this recipe can work for music as well: the winning DORA-MOS system achieved the highest correlation for musical impression prediction. However, DORA-MOS and other top systems remain closed-source, preventing the community from building on or reproducing these results. The quality assessment bottleneck therefore persists not because a solution is impossible, but because no open, reproducible solution exists.
This paper asks: can the “deep features + quality annotations” recipe produce an open, reproducible per-sample quality metric for generated music, and which components of the training pipeline actually matter? We systematically investigate this question through a progressive ablation study on MusicEval [13], a recently released dataset of 2,748 generated music clips from 31 TTM systems with 13,740 expert quality ratings.
Our investigation yields three principal findings:
1. Frozen MuQ features are remarkably effective. A simple baseline consisting of frozen MuQ-310M [14] features with attention pooling and a two-layer MLP achieves system-level SRCC $=0.957$ and utterance-level SRCC $=0.838$ with human MOS. This approaches the closed-source DORA-MOS benchmark while being fully open and reproducible.
1. Progressive training complexity does not help. A systematic ablation over four training enhancements, namely ordinal classification loss ( $Δ=-0.004$ ), LoRA encoder adaptation ( $Δ=+0.007$ ), pairwise contrastive auxiliary loss ( $Δ=-0.004$ ), and uncertainty-weighted multi-task learning ( $Δ=-0.000$ ), shows that none improves system-level SRCC by the pre-registered threshold of $Δ≥ 0.02$ . The hypothesized cumulative improvement does not materialize.
1. Encoder choice is the dominant factor. Replacing MuQ-310M with MERT-95M [15] reduces system-level SRCC from 0.957 to 0.946, making encoder selection the single most impactful design decision.
1. Remarkably few annotations are needed. A data efficiency analysis shows that LoRA-adapted models reach utterance-level SRCC $=0.761$ with only 150 training clips, and match the frozen baseline’s performance at 500 clips using only 250 clips, approximately half the data. This makes personalized quality evaluators practical: a single listener annotating a collection comparable to one artist’s discography ( ${∼}150$ songs, e.g., Jay Chou’s 15 studio albums) could train a quality evaluator tailored to their aesthetic preferences. Because the annotations are on familiar real music, the resulting evaluator can then be applied to score generated music in that style—enabling an “annotate one discography, evaluate all generations” workflow for style-specific quality control.
We release MuQ-Eval as a fully open-source per-sample quality metric for generated music, including model weights, training code, and a standardized evaluation benchmark. The entire pipeline trains in under 2 GPU-hours on a single consumer-grade NVIDIA RTX 4080 (16 GB), a general-purpose GPU widely available to individual researchers. To our knowledge, this is the first open-source learned metric achieving system-level SRCC $>0.95$ with expert MOS ratings on music generation quality.
The remainder of this paper is organized as follows. Section II reviews existing evaluation metrics and related learned metrics from adjacent domains. Section III describes the model architecture and training pipeline. Section IV presents the experimental setup and ablation design. Section V reports results with statistical analysis. Section VI discusses implications and limitations. Section VII concludes.
## II Related Work
### II-A Distributional Metrics for Music Generation
We refer the reader to Kader et al. [16] for a comprehensive survey of music generation evaluation metrics; here we summarize the most relevant lines of work.
Fréchet Audio Distance (FAD) [4] adapts Fréchet Inception Distance to audio by computing the distance between Gaussian-fitted embedding distributions of generated and reference sets. FAD requires large sample sets and cannot score individual clips. Its correlation with human preferences is embedding-dependent: VGGish embeddings yield $τ=0.14$ [5], while PANNs embeddings yield $ρ>0.5$ on environmental audio [17]. Gui et al. [18] adapt FAD specifically for generative music evaluation and highlight the sensitivity of FAD rankings to embedding choice. Grotschla et al. [19] benchmark multiple metrics against human preference ratings and find that all tested metrics misranked at least one system (notably Riffusion), underscoring the fragility of current distributional metrics. Kernel Audio Distance (KAD) [20] replaces the Gaussian assumption with maximum mean discrepancy, achieving $ρ≈-0.80$ with PANNs-WGLM embeddings on DCASE data. Music-Aligned Distance (MAD) [5] uses MERT embeddings and reaches $τ=0.62$ with human preferences, the highest reported for a distributional metric on music, yet still substantially below speech-domain per-sample metrics.
### II-B Text-Audio Alignment Metrics
CLAP score [6, 7] computes cosine similarity between text and audio embeddings from contrastive language-audio pre-training. It measures semantic relevance rather than perceptual quality: a clip can achieve high CLAP score while sounding degraded. Human-CLAP [21] improves relevance correlation from $ρ≈ 0.26$ to $>0.50$ by integrating human feedback, but does not address quality assessment.
### II-C Per-Sample Learned Quality Metrics
#### Adjacent domains.
Learned perceptual metrics have been highly successful in vision and speech. LPIPS [10] uses linear probes on deep image features trained with 484k human judgments, achieving 68.9% 2AFC agreement (vs. 63.1% for SSIM, 73.9% human ceiling). In speech, NISQA [11] achieves PCC $=0.80$ – $0.95$ for MOS prediction, and DNSMOS [12] reaches PCC $=0.94$ – $0.98$ . More recently, SCOREQ [22] uses contrastive regression with speech self-supervised learning (SSL) features to achieve state-of-the-art domain generalization, and ALLD [23] leverages audio large language models for descriptive quality evaluation with SRCC $=0.93$ .
#### Music domain.
Per-sample quality metrics for music generation remain underdeveloped. PAM [24] prompts frozen audio language models for quality assessment without quality-specific training. Audiobox Aesthetics [25, 8] trains prediction heads on WavLM-Large features using 562 hours of multi-domain audio with aesthetic ratings, but achieves only Spearman $r=0.200$ on music generation preferences, likely because WavLM is a speech-domain encoder not optimized for music content. The AudioMOS 2025 Challenge [9] demonstrated that music quality prediction is feasible: the winning DORA-MOS system achieved SRCC $=0.988$ for musical impression using the MuQ encoder, but the system architecture and training details are not publicly available.
### II-D Music Understanding Encoders
MERT [15] is a music SSL model trained with RVQ-VAE and CQT teacher signals on 160K hours of music, achieving strong performance on the MARBLE benchmark. MuQ [14] uses Mel-RVQ tokenization and achieves higher MARBLE scores (77.0) with $100×$ less pre-training data, suggesting more efficient music representation learning. MuQ’s use in the DORA-MOS system provides evidence for its suitability as a quality prediction backbone.
### II-E Quality Annotation Datasets
MusicEval [13] provides 2,748 generated clips from 31 TTM systems with 13,740 expert ratings on musical impression (MI) and textual alignment (TA) using a 1–5 Likert scale. SongEval [26] offers 2,399 full-length songs with $∼$ 48k ratings across 5 aesthetic dimensions from 16 annotators. Both datasets are an order of magnitude smaller than speech equivalents (NISQA: 73k files; DNSMOS: $∼$ 150k annotations). However, as we show in Section V-H, the small dataset size may be less of a limitation than it appears: frozen encoder features can reach high correlation with only ${∼}1{,}500$ training samples, a volume that is within reach of a single listener’s annotation effort.
## III Method
### III-A Architecture
MuQ-Eval follows a simple encoder-pooling-head architecture. Given a music audio waveform $x∈ℝ^T$ at 24 kHz, we extract frame-level features $H=[h_1,…,h_L]∈ℝ^L× d$ from the last hidden layer of a pre-trained encoder, where $L$ is the number of frames and $d$ is the hidden dimension.
#### Encoder.
We use MuQ-310M [14] (OpenMuQ/MuQ-large-msd-iter), a Wav2Vec2-Conformer model with 24 layers and $d=1024$ , pre-trained for music understanding. The encoder processes 24 kHz input and produces features at 50 Hz frame rate.
#### Pooling.
Frame-level features are aggregated via learned attention pooling:
$$
z=∑_l=1^Lα_lh_l, α_l=\frac{\exp(w^⊤h_l)}{∑_l^\prime\exp(w^⊤h_l^\prime)} \tag{1}
$$
where $w∈ℝ^d$ is a learnable attention vector.
#### Prediction heads.
The pooled representation $z$ is passed to separate prediction heads for each quality dimension. Each head is a 2-layer MLP with GELU activation and hidden dimension 256:
$$
\hat{y}=MLP(z)=W_2·GELU(W_1z+b_1)+b_2 \tag{2}
$$
### III-B Training Variants
We investigate a progressive sequence of training configurations to isolate the contribution of each component:
#### A1: Frozen encoder + MSE (baseline).
The encoder is frozen; only the attention pooling and MLP heads are trained ( $∼$ 1M parameters). The loss is mean squared error between predicted and mean human MOS:
$$
L_MSE=\frac{1}{N}∑_i(\hat{y}_i-y_i)^2 \tag{3}
$$
#### A2: Frozen encoder + ordinal classification.
Following DORA-MOS [9], we replace MSE with Gaussian-softened ordinal cross-entropy. The MOS range [1, 5] is divided into $K=5$ bins. The soft target distribution for MOS value $y$ is:
$$
p_k(y)∝\exp≤ft(-\frac{(y-c_k)^2}{2σ^2}\right) \tag{4}
$$
where $c_k$ is the center of bin $k$ and $σ=0.5$ . The loss is:
$$
L_ord=-∑_kp_k(y)\log\hat{p}_k \tag{5}
$$
#### A3a: LoRA encoder adaptation.
We add LoRA [27] adapters to all attention projections (Q, K, V, O) with rank $r=16$ and $α=32$ , adding $∼$ 2M trainable parameters (0.6% of 310M). The encoder is no longer frozen.
#### A3b: + Contrastive auxiliary loss.
We add a pairwise contrastive loss encouraging the model to produce larger score differences for pairs with larger MOS differences. For pairs $(i,j)$ with $y_i>y_j+m$ (margin $m=0.5$ ):
$$
L_con=\max(0,-(\hat{y}_i-\hat{y}_j)+m) \tag{6}
$$
The total loss becomes:
$$
L=L_ord+0.5·L_con \tag{7}
$$
with the contrastive term warm-started at epoch 6.
#### A3c: + Uncertainty weighting.
Following Kendall et al. [28], each head learns a log-variance $\logσ_h^2$ that scales its loss contribution:
$$
L_total=∑_h\frac{1}{2σ_h^2}L_h+\logσ_h \tag{8}
$$
#### A4: MERT-95M encoder ablation.
We replace MuQ-310M with MERT-95M [15] (m-a-p/MERT-v1-95M) using full fine-tuning (no LoRA) with ordinal classification loss, to isolate the effect of encoder choice.
## IV Experimental Setup
### IV-A Dataset
We train and evaluate on MusicEval [13], consisting of 2,748 clips from 31 text-to-music systems. Each clip has expert ratings on musical impression (MI, overall quality) and textual alignment (TA) on a 1–5 Likert scale, with 5 ratings per clip (13,740 total). We use the mean rating per clip as the target score. Under 5-fold CV, each fold uses ${∼}1{,}540$ clips for training and ${∼}385$ for testing.
### IV-B Evaluation Protocol
We use 5-fold cross-validation stratified by TTM model to ensure that all 31 systems appear in each fold’s test set. Each fold has $∼$ 384–385 test clips. We report two levels of evaluation:
#### System-level.
For each of the 31 TTM models, we compute the mean predicted score and mean human MOS across all test clips from that model, then compute Spearman rank correlation (SRCC), Pearson correlation (PCC), and Kendall’s $τ$ between the 31-dimensional vectors of model means.
#### Utterance-level.
We compute PCC and SRCC between per-clip predicted scores and per-clip mean human MOS on each fold’s test set ( $∼$ 384 clips).
All correlations are reported as 5-fold means $±$ standard deviation with 95% bias-corrected and accelerated (BCa) bootstrap confidence intervals ( $B=1000$ , seed = 42).
### IV-C Statistical Testing
Pairwise comparisons between models use the Steiger test for dependent correlations with Bonferroni correction ( $α_adj=0.01$ for 5 comparisons). Ablation deltas are assessed against a pre-registered threshold of $Δ≥ 0.02$ (system-level SRCC) with effect sizes reported as Cohen’s $q$ .
### IV-D Baselines
#### FAD (VGGish).
Computed on each fold’s test set using the frechet-audio-distance library with VGGish embeddings. FAD is a distributional (set-level) metric and cannot be directly compared via per-sample correlation.
#### Audiobox Aesthetics.
We reference the published correlation $r=0.200$ with music generation human preferences from Zhang et al. [8], as the model is not publicly available for evaluation on MusicEval.
### IV-E Training Details
All models are trained with AdamW (learning rate $3× 10^-4$ for heads, $1× 10^-5$ for LoRA parameters), batch size 16, for up to 50 epochs with early stopping (patience 10, monitored on validation SRCC). Mixed precision (bf16) training on a single NVIDIA RTX 4080 (16 GB), a general-purpose consumer GPU. Audio inputs are resampled to 24 kHz and truncated/padded to 10 seconds. Gradient clipping at max norm 1.0. Training curves are shown in Figure 1.
<details>
<summary>2603.22677v1/x1.png Details</summary>
### Visual Description
## Line Graphs: Validation SRCC (MI) and Training Loss Over Epochs
### Overview
The image contains two line graphs comparing the performance of different model configurations (A1-A4) across training epochs. The left graph tracks **Validation SRCC (MI)** (Spearman Rank Correlation Coefficient) from 0.4 to 1.0, while the right graph tracks **Training Loss** from 0 to 6. Both graphs span 20 epochs (0–20) and include a reference line labeled "Gate (0.80)".
---
### Components/Axes
#### Left Graph (Validation SRCC)
- **X-axis**: Epoch (0–20, linear scale)
- **Y-axis**: Validation SRCC (MI) (0.4–1.0, linear scale)
- **Legend**:
- A1 (Frozen+MSE*): Orange
- A2 (Frozen+OrdCE): Blue
- A3a (+LoRA): Green
- A3b (+Contrastive): Yellow
- A3c (+Uncertainty): Dark Blue
- A4 (MERT-95M): Red (dashed)
- Gate (0.80): Gray dashed horizontal line
#### Right Graph (Training Loss)
- **X-axis**: Epoch (0–20, linear scale)
- **Y-axis**: Training Loss (0–6, linear scale)
- **Legend**: Same as left graph (colors match corresponding lines).
---
### Detailed Analysis
#### Left Graph (Validation SRCC)
- **A1 (Frozen+MSE*)**: Starts at ~0.75, peaks at ~0.85 (epoch 5), then fluctuates between 0.82–0.85.
- **A2 (Frozen+OrdCE)**: Similar to A1 but slightly lower, peaking at ~0.83 (epoch 5), then stabilizes near 0.82.
- **A3a (+LoRA)**: Peaks at ~0.84 (epoch 5), dips to ~0.81 (epoch 15), then recovers to ~0.83.
- **A3b (+Contrastive)**: Starts at ~0.78, rises to ~0.83 (epoch 5), then declines to ~0.81 (epoch 20).
- **A3c (+Uncertainty)**: Most stable, fluctuates between 0.82–0.84.
- **A4 (MERT-95M)**: Consistently lowest, hovering between 0.75–0.78.
- **Gate (0.80)**: Horizontal line at 0.80; all models except A4 exceed this threshold.
#### Right Graph (Training Loss)
- **A1 (Frozen+MSE*)**: Starts at ~6, drops sharply to ~0.5 (epoch 2), then plateaus at ~0.4.
- **A2 (Frozen+OrdCE)**: Starts at ~2.5, decreases gradually to ~1.8 (epoch 20).
- **A3a (+LoRA)**: Starts at ~2.2, decreases to ~1.9 (epoch 20).
- **A3b (+Contrastive)**: Starts at ~2.1, decreases to ~1.7 (epoch 20).
- **A3c (+Uncertainty)**: Starts at ~2.0, decreases to ~1.6 (epoch 20).
- **A4 (MERT-95M)**: Lowest, starts at ~0.8, decreases to ~0.3 (epoch 20).
- **Gate (0.80)**: Horizontal line at 0.80; only A4 remains below this threshold after epoch 5.
---
### Key Observations
1. **Validation Performance**:
- A1 and A2 achieve the highest SRCC, surpassing the Gate threshold (0.80) consistently.
- A4 underperforms, remaining below the Gate line.
2. **Training Loss**:
- A1 experiences a dramatic initial drop but stabilizes at a low loss (~0.4).
- A4 has the lowest training loss throughout, suggesting efficient learning.
3. **Divergence Between Metrics**:
- A1 has high validation SRCC but also high training loss initially, indicating potential overfitting.
- A4 has low training loss but poor validation performance, suggesting underfitting or misalignment with the validation metric.
---
### Interpretation
- **Model Effectiveness**:
- A1 and A2 balance high validation performance with moderate training loss, making them strong candidates.
- A3 variants show moderate performance, with A3c being the most stable.
- A4’s low training loss but poor validation SRCC highlights a mismatch between training and validation objectives.
- **Gate Threshold**:
- The Gate line (0.80) serves as a performance benchmark. Models above it (A1-A3) meet or exceed this standard, while A4 falls short.
- **Training Dynamics**:
- A1’s sharp initial loss reduction suggests aggressive optimization, but its plateau at ~0.4 loss may indicate saturation.
- A4’s consistently low loss implies efficient training but insufficient capacity to capture validation complexity.
---
### Spatial Grounding & Verification
- **Legend Placement**: Right side of both graphs, ensuring clear association with line colors.
- **Color Consistency**: Confirmed matches between legend labels and line colors (e.g., A1 = orange, A4 = red).
- **Axis Alignment**: Both graphs use linear scales for epochs (0–20) and their respective metrics.
---
### Content Details
- **Validation SRCC (MI)**:
- A1: Peaks at 0.85 (epoch 5), ends at 0.83.
- A2: Peaks at 0.83 (epoch 5), ends at 0.82.
- A3a: Peaks at 0.84 (epoch 5), ends at 0.83.
- A3b: Peaks at 0.83 (epoch 5), ends at 0.81.
- A3c: Stable at 0.82–0.84.
- A4: 0.75–0.78.
- **Training Loss**:
- A1: Drops from 6 to 0.4 (epoch 2–20).
- A2: Drops from 2.5 to 1.8.
- A3a: Drops from 2.2 to 1.9.
- A3b: Drops from 2.1 to 1.7.
- A3c: Drops from 2.0 to 1.6.
- A4: Drops from 0.8 to 0.3.
---
### Notable Anomalies
- **A1’s Training Loss**: Unusually high initial value (~6) compared to other models, suggesting a different optimization trajectory or data scaling issue.
- **A4’s Validation SRCC**: Persistently below the Gate threshold despite low training loss, indicating potential underfitting or metric misalignment.
</details>
Figure 1: Training dynamics: (a) validation SRCC vs. epoch and (b) training loss vs. epoch for all configurations (fold 0). All MuQ variants converge to similar validation SRCC.
## V Results
### V-A Main Results: System-Level Correlation
Table I presents system-level correlations with human MOS across 31 TTM models. All MuQ-based variants achieve SRCC $>0.95$ , substantially exceeding the pre-registered target of 0.90 and the published Audiobox Aesthetics correlation of $r=0.200$ .
TABLE I: System-level correlation with human MOS (31 TTM models, MusicEval 5-fold CV). Best in bold; 95% BCa bootstrap CIs in brackets.
| Method FAD (VGGish) † Audiobox Aes. ∗ | SRCC ${}_sys$ 0.295 $±$ .031 ‡ 0.200 | PCC ${}_sys$ — — | $τ_sys$ 0.14 — |
| --- | --- | --- | --- |
| A1 (Frozen+MSE) | 0.957 $±$ .014 | 0.960 $±$ .013 | 0.839 $±$ .019 |
| [.898, .986] | [.916, .980] | [.716, .910] | |
| A2 (Frozen+Ord.) | 0.953 $±$ .015 | 0.963 $±$ .007 | 0.839 $±$ .034 |
| [.873, .986] | [.912, .982] | [.696, .915] | |
| A3a (+LoRA) | 0.960 $±$ .017 | 0.969 $±$ .009 | 0.846 $±$ .034 |
| [.902, .987] | [.937, .984] | [.715, .913] | |
| A3b (+Contr.) | 0.955 $±$ .016 | 0.967 $±$ .010 | 0.838 $±$ .035 |
| [.892, .984] | [.935, .981] | [.702, .908] | |
| A3c (+Unc.Wt.) | 0.955 $±$ .016 | 0.968 $±$ .010 | 0.836 $±$ .035 |
| [.893, .984] | [.935, .981] | [.704, .907] | |
| A4 (MERT-95M) | 0.946 $±$ .012 | 0.948 $±$ .013 | 0.822 $±$ .022 |
| [.880, .981] | [.906, .971] | [.695, .898] | |
The simplest model, A1 (frozen MuQ + MSE), achieves SRCC $=0.957$ [0.898, 0.986], nearly matching the highest point estimate (A3a: 0.960). The Steiger test confirms no significant difference between A1 and any other MuQ variant at the Bonferroni-corrected threshold ( $p>0.01$ for all pairwise comparisons with A1). Figure 2 visualizes the correspondence between predicted and human MOS at the system level.
<details>
<summary>2603.22677v1/x2.png Details</summary>
### Visual Description
## Scatter Plot: A3c: System-Level (Fold 0, N=31 systems)
### Overview
The image is a scatter plot comparing **Predicted Mean MOS (per system)** (y-axis) to **Human Mean MOS (per system)** (x-axis). A dashed trend line indicates a strong positive correlation, with high SRCC (0.966) and PCC (0.972) values. Data points are represented as blue dots.
### Components/Axes
- **Title**: "A3c: System-Level (Fold 0, N=31 systems)"
- **X-axis**:
- Label: "Human Mean MOS (per system)"
- Scale: 2.5 to 4.5 (increments of 0.5)
- **Y-axis**:
- Label: "Predicted Mean MOS (per system)"
- Scale: 2.5 to 4.5 (increments of 0.5)
- **Trend Line**: Dashed gray line with slope ≈1, intercept ≈2.3.
- **Data Points**: Blue dots (no explicit legend, but color matches implied legend).
### Detailed Analysis
- **Data Points**:
- **X-axis (Human Mean MOS)**: Ranges from 2.5 to 4.5.
- **Y-axis (Predicted Mean MOS)**: Ranges from 2.5 to 4.5.
- Points cluster tightly around the dashed trend line, with minimal deviation.
- Notable outliers:
- One point at (2.5, 2.6) deviates slightly below the trend line.
- Two points at (4.3, 4.3) and (4.4, 4.3) cluster near the upper-right corner.
- **Correlation Metrics**:
- SRCC = 0.966 (Spearman’s Rank Correlation Coefficient)
- PCC = 0.972 (Pearson Correlation Coefficient)
### Key Observations
1. **Strong Linear Relationship**: The high SRCC and PCC values confirm a near-perfect positive correlation between predicted and human Mean MOS.
2. **Model Accuracy**: Predicted values closely align with human evaluations, especially in the mid-range (3.0–4.0).
3. **Outliers**: Minimal deviation from the trend line, suggesting robust model performance.
### Interpretation
The data demonstrates that the A3c model’s predictions for system-level Mean Opinion Scores (MOS) are highly accurate, with a correlation coefficient exceeding 0.96. This implies the model effectively captures human-like judgments, likely due to its architecture (e.g., reinforcement learning with human feedback). The slight spread in data points suggests minor variability in predictions, but overall, the model’s reliability is exceptional. The trend line’s slope ≈1 indicates a 1:1 relationship, meaning predictions scale proportionally with human evaluations.
</details>
Figure 2: System-level scatter: predicted MuQ-Eval scores vs. human MOS for 31 TTM models (A1, fold 0). Dashed: linear fit; dotted: $y=x$ reference.
FAD (VGGish) computed on the same test folds yields a mean of $0.295± 0.031$ across folds. As a distributional metric, FAD does not produce per-sample scores and thus cannot be directly compared via correlation with human MOS. However, published analyses report FAD (VGGish) correlation with human preferences at $τ=0.14$ [5], approximately $6×$ lower than MuQ-Eval ’s system-level $τ=0.839$ .
### V-B Utterance-Level Correlation
Table II shows per-clip correlations on the musical impression (MI) dimension. All models achieve utterance-level PCC $>0.80$ and SRCC $>0.82$ , exceeding the pre-registered targets of 0.70 and 0.65, respectively.
TABLE II: Utterance-level correlation with human MOS (MI dimension, $∼$ 385 clips/fold, 5-fold CV). 95% BCa CIs in brackets.
| A1 A2 A3a | .828 $±$ .017 .824 $±$ .020 .827 $±$ .020 | .838 $±$ .016 .833 $±$ .019 .835 $±$ .019 | .594 $±$ .041 .604 $±$ .024 .614 $±$ .029 | .591 $±$ .034 .601 $±$ .023 .614 $±$ .029 |
| --- | --- | --- | --- | --- |
| A3b | .829 $±$ .019 | .840 $±$ .019 | .588 $±$ .036 | .585 $±$ .034 |
| A3c | .829 $±$ .019 | .840 $±$ .019 | .587 $±$ .038 | .584 $±$ .035 |
| A4 | .808 $±$ .019 | .825 $±$ .015 | .613 $±$ .034 | .619 $±$ .028 |
The MI head substantially outperforms the TA head across all configurations (SRCC 0.83–0.84 vs. 0.58–0.62), suggesting that textual alignment is a harder prediction target that may require text-conditioned architectures. Figure 3 shows the per-clip prediction quality.
<details>
<summary>2603.22677v1/x3.png Details</summary>
### Visual Description
## Scatter Plot: A3c Utterance-Level Prediction (Fold 0, N=385)
### Overview
The image is a scatter plot comparing human-assessed Mean Opinion Scores (MOS) against model-predicted MOS values for utterance-level analysis. The plot includes a dashed line representing the line of perfect prediction (y=x), with statistical metrics (PCC=0.828, SRCC=0.838) indicating strong correlation between human and model assessments.
### Components/Axes
- **X-axis**: Human MOS (MI)
- Scale: 1.0 to 5.0 (linear)
- Label: "Human MOS (MI)"
- **Y-axis**: Predicted MOS (MI)
- Scale: 1.0 to 5.0 (linear)
- Label: "Predicted MOS (MI)"
- **Dashed Line**: Gray line of perfect prediction (y=x)
- **Data Points**: Blue circles (N=385)
- **Title**: "A3c: Utterance-Level (Fold 0, N=385)"
- **Statistical Metrics**:
- PCC (Pearson Correlation Coefficient): 0.828
- SRCC (Spearman Rank Correlation Coefficient): 0.838
### Detailed Analysis
- **Data Distribution**:
- Points cluster tightly around the dashed line, with minor vertical/horizontal spread.
- Higher density of points in the 2.5–4.5 MOS range for both axes.
- No extreme outliers (all points within 1.0–5.0 bounds).
- **Correlation Metrics**:
- PCC=0.828 (strong linear relationship)
- SRCC=0.838 (strong monotonic relationship)
- **Fold Context**: "Fold 0" suggests this is one iteration of cross-validation (likely part of a larger k-fold analysis).
### Key Observations
1. **High Agreement**: The near-diagonal clustering of points indicates the model predictions closely match human judgments.
2. **Consistent Correlation**: Both PCC and SRCC values exceed 0.8, suggesting robust performance across linear and rank-based relationships.
3. **Uniform Coverage**: Data points span the full MOS range (1.0–5.0), though mid-range scores dominate.
### Interpretation
The plot demonstrates that the A3c model achieves high fidelity in predicting human-rated utterance quality. The strong PCC/SRCC values confirm the model captures both magnitude and ordinal relationships in MOS data. The "Fold 0" designation implies this is one validation iteration, and similar results across other folds would strengthen confidence in generalizability. The MI unit (likely Mel scale) contextualizes the scores as perceptual ratings, emphasizing the model's utility in speech/utterance evaluation tasks. The absence of extreme outliers suggests the model handles edge cases (e.g., very low/high scores) reasonably well.
</details>
Figure 3: Utterance-level scatter: per-clip predicted vs. human MOS (MI dimension, A1, fold 0, $∼$ 385 clips). Higher density along the diagonal indicates strong per-sample agreement.
### V-C Ablation Analysis: Progressive Training Recipe
Table III presents the progressive ablation results. No component achieves the pre-registered $Δ≥ 0.02$ threshold at system level.
TABLE III: Progressive ablation deltas (system-level SRCC). Pre-registered threshold: $Δ≥ 0.02$ . None met.
| A2 $-$ A1 | Ordinal CE | $-0.004$ | 0.050 | No |
| --- | --- | --- | --- | --- |
| A3a $-$ A2 | LoRA tuning | $+0.007$ | 0.085 | No |
| A3b $-$ A3a | Contrastive | $-0.004$ | 0.054 | No |
| A3c $-$ A3b | Unc. weight | $-0.000$ | 0.003 | No |
#### Ordinal CE vs. MSE ( $Δ=-0.004$ ).
Gaussian-softened ordinal classification, used successfully in DORA-MOS [9], does not improve over simple MSE regression. Cohen’s $q=0.050$ indicates a negligible effect. At utterance level, ordinal CE shows a small improvement in TA correlation (+0.010 SRCC) but this is offset by a decrease in MI correlation ( $-0.005$ SRCC).
#### LoRA adaptation ( $Δ=+0.007$ ).
Fine-tuning the encoder with LoRA produces the largest positive delta but remains below threshold. The effect is small (Cohen’s $q=0.085$ ) and adds 2M trainable parameters and training complexity.
#### Contrastive loss ( $Δ=-0.004$ ).
Adding the pairwise contrastive objective slightly degrades system-level performance. At utterance level, it creates a trade-off: MI SRCC increases by +0.005 (A3b vs. A3a) but TA SRCC decreases by $-0.029$ , suggesting the contrastive signal conflicts with the textual alignment task.
#### Uncertainty weighting ( $Δ=-0.000$ ).
Learned uncertainty weighting has no measurable effect, with Cohen’s $q=0.003$ , functionally zero.
Figure 4 summarizes the ablation visually, showing both system-level and utterance-level SRCC(MI) side by side. All configurations cluster tightly above the 0.90 target at system level with overlapping confidence intervals, and utterance-level correlations follow a similar pattern with narrower spread.
<details>
<summary>2603.22677v1/x4.png Details</summary>
### Visual Description
## Bar Chart: SRCC (MI) Performance Across Phases and System/Utterance Levels
### Overview
The chart compares the Sentence-Level Recall and Coherence (SRCC) metric (measured in MI) across four phases (A1-A4) and two system levels (system-level and utterance-level). System-level performance is represented by solid bars, while utterance-level performance uses hatched bars. Error bars indicate measurement uncertainty.
### Components/Axes
- **Y-axis**: SRCC (MI) values ranging from 0.60 to 1.00 in 0.05 increments.
- **X-axis**: Categories labeled A1, A2, A3a, A3b, A3c, and A4.
- **Legend**:
- Blue: Phase 1: Baselines
- Green: Phase 2: Ablation
- Orange: Phase 3: Encoder
- Solid: System-level (solid bars)
- Hatched: Utterance-level (hatched bars)
### Detailed Analysis
1. **Phase 1 (Baselines, Blue)**:
- **A1**: System-level = 0.957 (±0.005), Utterance-level = 0.838 (±0.005)
- **A2**: System-level = 0.953 (±0.005), Utterance-level = 0.833 (±0.005)
2. **Phase 2 (Ablation, Green)**:
- **A3a**: System-level = 0.960 (±0.005), Utterance-level = 0.835 (±0.005)
- **A3b**: System-level = 0.955 (±0.005), Utterance-level = 0.840 (±0.005)
- **A3c**: System-level = 0.955 (±0.005), Utterance-level = 0.840 (±0.005)
3. **Phase 3 (Encoder, Orange)**:
- **A4**: System-level = 0.946 (±0.005), Utterance-level = 0.825 (±0.005)
### Key Observations
- **System-level dominance**: Solid bars (system-level) consistently outperform hatched bars (utterance-level) across all categories, with a maximum gap of ~0.12 MI (A1).
- **Phase 2 peak**: The highest system-level performance (0.960 MI) occurs in A3a (Phase 2), suggesting ablation in this phase optimizes performance.
- **Utterance-level plateau**: Utterance-level values stabilize at 0.840 MI in A3b/A3c (Phase 2) before dropping to 0.825 MI in A4 (Phase 3).
- **Phase 3 regression**: System-level performance declines from 0.955 MI (A3c) to 0.946 MI (A4), indicating potential degradation in the encoder phase.
### Interpretation
The data demonstrates that system-level SRCC consistently exceeds utterance-level performance, emphasizing the importance of holistic system design over isolated utterance optimization. The peak in Phase 2 (A3a) suggests ablation techniques in this phase yield optimal results, while the Phase 3 decline may indicate over-optimization or architectural trade-offs. The stable utterance-level performance across A3b/A3c implies that utterance-specific improvements plateau early, leaving system-level adjustments as the primary lever for enhancement. The error bars (all ±0.005) indicate high measurement precision, reinforcing the reliability of these trends.
</details>
Figure 4: Progressive ablation: system-level (solid bars) and utterance-level (hatched bars) SRCC(MI) per experiment. Trend lines connect values across experiments. All MuQ variants cluster tightly at both granularities, confirming that progressive complexity does not yield cumulative improvement.
### V-D Encoder Comparison
Replacing MuQ-310M with MERT-95M (experiment A4) yields the largest observed drop in system-level SRCC. Relative to the frozen baseline A1, the delta is $Δ=-0.011$ (0.957 vs. 0.946); relative to A3c, $Δ=-0.009$ with Cohen’s $q=0.192$ , although the Steiger test does not reach significance after Bonferroni correction ( $p=0.200$ ). MERT-95M uses full fine-tuning with 95M trainable parameters versus MuQ’s $∼$ 1M frozen parameters, yet achieves lower correlation, reinforcing that MuQ’s pre-training provides superior quality-relevant representations for this task.
### V-E Statistical Comparisons
Table IV summarizes pairwise Steiger tests against the best-performing variant at the highest system-level SRCC (A3a, 0.960) and the designated final model (A3c, 0.955).
TABLE IV: Steiger test for dependent correlations (system-level SRCC). Bonferroni $α_adj=0.01$ .
| A3c vs. A1 A3c vs. A2 A3c vs. A3a | $-0.485$ $1.466$ $-3.714$ | 0.284 0.002 0.009 | 0.074 0.109 0.084 | No Yes Yes |
| --- | --- | --- | --- | --- |
| A3c vs. A3b | $-0.766$ | 0.496 | 0.005 | No |
| A3c vs. A4 | $2.732$ | 0.200 | 0.192 | No |
Two comparisons reach significance: A3c outperforms A2 ( $p=0.002$ ) and A3a outperforms A3c ( $p=0.009$ ). Notably, the more complex A3c is significantly worse than the intermediate A3a at system level. We caution that the Steiger test on $n=31$ systems with high within-fold correlation may inflate significance; all effect sizes (Cohen’s $q$ ) remain in the “small” range ( $<0.30$ ).
### V-F Degradation Concordance
To assess whether the metric produces monotonically decreasing scores for progressively degraded audio, we apply four types of controlled degradation (MP3 compression, additive Gaussian noise, pitch shift, and tempo stretch) at three severity levels to the top-quartile quality clips ( $N=114$ ) from fold 0’s test set. We measure concordance: the fraction of clip pairs where the undegraded original scores higher than its degraded counterpart.
TABLE V: Degradation concordance (A3b, fold 0, $N=114$ clips). Values are the fraction of pairs where the original scores higher than the degraded version. Chance level is 0.50.
| MP3 compression Gaussian noise Pitch shift | 0.97 0.99 0.51 | 0.84 0.89 0.43 | 0.52 0.39 0.43 |
| --- | --- | --- | --- |
| Tempo stretch | 0.58 | 0.49 | 0.46 |
Table V and Figure 5 reveal a clear dichotomy. The metric reliably detects signal-level artifacts: MP3 compression at 32 kbps (concordance $=0.97$ ) and additive noise at SNR 10 dB ( $0.99$ ) produce near-perfect detection. However, musical-structural distortions, specifically pitch shift ( $0.43$ – $0.51$ ) and tempo stretch ( $0.46$ – $0.58$ ), yield concordance near chance at all severities. The overall mean concordance is $0.63$ , failing the pre-registered target of $0.85$ .
This selective sensitivity is consistent with MuQ’s pre-training, which likely normalizes pitch and tempo variation (common augmentations in music understanding tasks) while remaining sensitive to spectral corruption. Mild degradations are near or below chance across all types, suggesting that the metric’s quality sensitivity has a threshold below which artifacts are not reliably detected.
<details>
<summary>2603.22677v1/x5.png Details</summary>
### Visual Description
## Heatmap: Concordance Rate by Audio Degradation Technique and Severity
### Overview
The image is a heatmap visualizing concordance rates (0.0–1.0) for four audio processing techniques across three degradation severity levels: Severe, Moderate, and Mild. Darker purple shades represent higher concordance rates, while lighter shades indicate lower rates.
### Components/Axes
- **X-axis (Degradation Severity)**: Labeled "Severe" (left), "Moderate" (center), and "Mild" (right).
- **Y-axis (Audio Techniques)**: Labeled "MP3 compression," "Gaussian noise," "Pitch shift," and "Tempo stretch."
- **Legend**: A vertical color bar on the right, ranging from 0.0 (white) to 1.0 (dark purple), labeled "Concordance Rate."
### Detailed Analysis
- **MP3 compression**:
- Severe: 0.97 (dark purple)
- Moderate: 0.84 (medium purple)
- Mild: 0.52 (light purple)
- **Gaussian noise**:
- Severe: 0.99 (darkest purple)
- Moderate: 0.89 (medium-dark purple)
- Mild: 0.39 (very light purple)
- **Pitch shift**:
- Severe: 0.51 (light purple)
- Moderate: 0.43 (pale purple)
- Mild: 0.43 (pale purple)
- **Tempo stretch**:
- Severe: 0.58 (medium-light purple)
- Moderate: 0.49 (pale purple)
- Mild: 0.46 (pale purple)
### Key Observations
1. **Highest concordance rates** are observed for **MP3 compression (0.97)** and **Gaussian noise (0.99)** under **Severe degradation**.
2. **Gaussian noise** shows the steepest drop in concordance rate (0.99 → 0.39) as degradation severity decreases from Severe to Mild.
3. **Pitch shift** and **Tempo stretch** exhibit consistently low concordance rates (<0.5) across all severity levels.
4. **MP3 compression** maintains the highest concordance rate in Mild degradation (0.52), outperforming other techniques except Gaussian noise in Severe/Moderate.
### Interpretation
The data suggests that **MP3 compression** and **Gaussian noise** are more robust to severe audio degradation, maintaining high concordance rates. However, their performance degrades significantly in milder conditions, which may indicate over-optimization for extreme cases. **Pitch shift** and **Tempo stretch** are less effective overall, with minimal improvement as degradation severity decreases. The anomaly of **Gaussian noise** having the lowest concordance rate in Mild degradation (0.39) warrants further investigation, as it contradicts the general trend of higher rates in less severe conditions. This heatmap highlights trade-offs between degradation resilience and performance consistency across techniques.
</details>
Figure 5: Degradation concordance heatmap (A3b, fold 0). The metric reliably detects signal-level artifacts (MP3, noise) at severe levels but is insensitive to musical-structural distortions (pitch, tempo) at all severities.
### V-G Computational Cost
Table VI summarizes computational requirements. The recommended A1 model requires only $∼$ 1M trainable parameters, $∼$ 3 GB peak VRAM, and $∼$ 35 ms inference per 10-second clip. All experiments were conducted on a single general-purpose consumer GPU (NVIDIA RTX 4080, 16 GB), widely accessible to individual researchers and small labs without dedicated compute clusters.
TABLE VI: Computational cost comparison.
| A1 A3a–c A4 | MuQ-310M MuQ-310M MERT-95M | $∼$ 1M $∼$ 2M 95M | $∼$ 3 GB $∼$ 4 GB $∼$ 12 GB | $∼$ 35 $∼$ 35 $∼$ 20 |
| --- | --- | --- | --- | --- |
### V-H Data Efficiency
To quantify how much training data is needed, we train both A1 (frozen MuQ + MSE) and A3a (LoRA + ordinal CE) on subsampled training sets ranging from 100 to 1,000 clips (fold 0, test set unchanged at ${∼}385$ clips). This comparison reveals the sample efficiency of each approach and the practical feasibility of building quality evaluators from small annotation budgets.
TABLE VII: Data efficiency: utterance-level MI correlation as a function of training set size (fold 0). A3a (LoRA) achieves higher correlation at every sample size and matches A1@500 performance with only 250 samples.
| 100 150 250 | 0.635 0.685 0.731 | 0.623 0.679 0.720 | 0.757 0.761 0.788 | 0.742 0.751 0.778 |
| --- | --- | --- | --- | --- |
| 500 | 0.781 | 0.763 | 0.808 | 0.799 |
| 750 | 0.814 | 0.804 | 0.823 | 0.815 |
| 1,000 | 0.836 | 0.829 | 0.824 | 0.809 |
| Full ( ${∼}$ 1,540) | 0.838 | 0.828 | 0.835 | 0.827 |
Table VII reveals two findings. First, A3a (LoRA) is consistently more sample-efficient than A1 (frozen): at $N=100$ , A3a achieves MI SRCC $=0.757$ versus A1’s $0.635$ (a gap of $0.122$ ). The gap narrows with more data, converging at the full training set. Second, a crossover exists: A3a with only 250 samples (SRCC $=0.788$ ) exceeds A1 with 500 samples (SRCC $=0.781$ ), meaning LoRA fine-tuning requires approximately half the training data to match the frozen baseline’s performance. This advantage likely arises because encoder adaptation captures quality-relevant features that are present but not linearly accessible in the frozen representations.
The practical implication is that personalized quality evaluators are feasible with very small annotation budgets. A3a trained on just 150 clips achieves MI SRCC $=0.761$ , already a practically useful correlation. To put 150 clips in perspective: a typical music album contains 10–14 tracks; a prolific artist like Jay Chou has ${∼}150$ songs across 15 studio albums (and ${∼}350$ including songs composed for other artists); a curated Spotify playlist averages 20–60 songs. A listener who rates a personal collection comparable to a few albums, roughly 5 hours of annotation effort at 30 clips/hour, could train a quality evaluator tailored to their own aesthetic preferences. Figure 6 visualizes the data efficiency curves for both models.
<details>
<summary>2603.22677v1/x6.png Details</summary>
### Visual Description
## Line Graphs: Validation Performance vs Training Samples
### Overview
Two line graphs compare the validation performance of two models (A1 and A3a) across increasing training samples. The left graph measures **Validation SRCC (MI)**, while the right graph measures **Validation PCC (MI)**. Both graphs show performance trends for two configurations:
- **A1 (Frozen + MSE)** (orange line with circles)
- **A3a (LoRA + Ordinal CE)** (green line with squares)
A green arrow highlights a key comparison: **A3a@250 > A1@500**.
---
### Components/Axes
- **X-axis (both graphs)**: "Training samples" (0 to 1600 in increments of 200).
- **Y-axis (left graph)**: "Validation SRCC (MI)" (0.55 to 0.90).
- **Y-axis (right graph)**: "Validation PCC (MI)" (0.55 to 0.90).
- **Legends**:
- **A1 (Frozen + MSE)**: Orange line with circular markers.
- **A3a (LoRA + Ordinal CE)**: Green line with square markers.
- **Note**: Green arrow labeled "A3a@250 > A1@500" points to the left graph.
---
### Detailed Analysis
#### Left Graph (Validation SRCC)
- **A1 (Frozen + MSE)**:
- Starts at **0.64** (200 samples).
- Increases steadily to **0.84** (1600 samples).
- Slope: Gradual upward trend.
- **A3a (LoRA + Ordinal CE)**:
- Starts at **0.76** (200 samples).
- Peaks at **0.83** (250 samples).
- Slight dip to **0.84** (1600 samples).
- Slope: Sharp initial rise, then plateaus.
#### Right Graph (Validation PCC)
- **A1 (Frozen + MSE)**:
- Starts at **0.62** (200 samples).
- Rises to **0.83** (1600 samples).
- Slope: Steady upward trend.
- **A3a (LoRA + Ordinal CE)**:
- Starts at **0.74** (200 samples).
- Peaks at **0.82** (800 samples).
- Slight increase to **0.83** (1600 samples).
- Slope: Rapid initial growth, then stabilizes.
---
### Key Observations
1. **Performance Trends**:
- Both models improve with more training samples.
- **A3a** consistently outperforms **A1** in early stages (e.g., 200–500 samples).
- **A1** catches up to **A3a** by 1600 samples in both metrics.
2. **Notable Comparison**:
- The green arrow highlights that **A3a at 250 samples (0.83 SRCC)** exceeds **A1 at 500 samples (0.77 SRCC)**.
- Similar pattern in PCC: **A3a at 250 samples (0.77)** vs. **A1 at 500 samples (0.72)**.
3. **Model Efficiency**:
- **A3a (LoRA + Ordinal CE)** achieves higher performance with fewer samples, suggesting better parameter efficiency.
- **A1 (Frozen + MSE)** requires more data to match **A3a**'s early performance.
---
### Interpretation
The data demonstrates that **A3a (LoRA + Ordinal CE)** is more sample-efficient than **A1 (Frozen + MSE)**, particularly in early training stages. While **A1** eventually matches **A3a**'s performance with sufficient data, the former’s reliance on larger datasets may limit practicality in resource-constrained scenarios. The use of **LoRA** (Low-Rank Adaptation) and **Ordinal CE** (Cross-Entropy) in **A3a** likely contributes to its faster convergence, whereas **A1**’s **Frozen** weights and **MSE** loss function may require more samples to optimize effectively. This suggests that architectural choices (e.g., LoRA) and loss function design (Ordinal CE) significantly impact model efficiency and scalability.
</details>
Figure 6: Data efficiency: utterance-level MI SRCC and PCC as a function of training set size. A3a (LoRA) consistently outperforms A1 (frozen) and requires ${∼}50\$ fewer samples to reach comparable performance (annotated crossover: A3a@250 $>$ A1@500).
## VI Discussion
### VI-A Why Do Frozen Features Suffice?
The finding that frozen MuQ features with a simple MLP match or exceed more sophisticated training recipes is consistent with the LPIPS result in vision [10], where linear probes on deep network features closely approach the human perceptual ceiling. We hypothesize that MuQ’s pre-training on music understanding tasks produces representations that already encode quality-relevant attributes (pitch accuracy, rhythmic coherence, timbral naturalness), and that the MOS prediction task on MusicEval’s quality range is “linearly separable” in this embedding space. The 31 TTM systems in MusicEval span a wide quality range (from poor to near-human), which may favor linear decodability. Whether frozen features remain sufficient for finer-grained quality discrimination (e.g., distinguishing between two high-quality systems) remains an open question.
### VI-B Data Efficiency and Personalized Evaluators
Our data efficiency analysis (Section V-H, Table VII) reveals that LoRA-adapted models are substantially more sample-efficient than frozen baselines. At $N=100$ , A3a achieves MI SRCC $=0.757$ versus A1’s $0.635$ , a gap of $0.122$ . This advantage persists across all sample sizes and narrows only at the full training set ( ${∼}1{,}540$ clips). The crossover point (A3a@250 $≈$ A1@500) demonstrates that encoder adaptation approximately halves the annotation requirement.
This opens the door to personalized or style-specific quality evaluators. The pre-trained encoder captures music-relevant attributes; LoRA adaptation then tunes these features for quality prediction with minimal supervision. In practice, this means a single listener who annotates a collection of ${∼}150$ familiar songs, comparable to one artist’s studio discography (e.g., Jay Chou’s ${∼}150$ songs across 15 studio albums, or ${∼}350$ including songs composed for other artists), could train a quality evaluator achieving SRCC $>0.76$ . With ${∼}500$ annotations, performance exceeds SRCC $=0.80$ .
Crucially, this personalization workflow is lightweight and tailorable to a specific target style. A user interested in evaluating generated music in a particular style—say, Mandopop ballads or lo-fi hip-hop—need only rate one representative artist’s discography in that style. The resulting evaluator then serves as a style-specific quality filter for generated outputs: it scores new clips according to the learned quality standard of that style, rejecting generations that deviate from the expected timbral, harmonic, or production characteristics. Because the annotation is done once on familiar, real music (not on generated outputs), the labeling effort is natural and efficient—a listener rates songs they already know well.
This “annotate one discography, evaluate all generations” paradigm has several practical applications. It could serve niche creative communities (e.g., lo-fi producers, classical composers, game audio designers) that are poorly served by generic quality metrics trained on broad expert consensus. It could enable preference-aligned generation pipelines where the reward signal comes from an individual’s taste rather than population-average MOS. And it could support A/B testing of generation models against a personal quality standard, allowing individual creators to select the model that best matches their artistic vision without running costly listening studies.
### VI-C Implications for Metric Design
Our negative ablation results have practical implications: practitioners building music quality metrics should invest in encoder selection rather than complex training objectives or fine-tuning strategies. The $5×$ difference in correlation attributable to encoder choice (MuQ vs. MERT, $Δ_q=0.192$ ) dwarfs any training recipe effect ( $q<0.09$ ). This is consistent with findings in speech quality prediction, where encoder pre-training quality dominates downstream task performance [22].
### VI-D System-Level vs. Utterance-Level Gap
All models show substantially higher system-level (SRCC $∼$ 0.95) than utterance-level (SRCC $∼$ 0.84) correlation. This gap reflects the noise reduction from averaging: system-level scores aggregate $∼$ 88 clips per TTM model ( $2748/31$ ), smoothing prediction errors. The utterance-level SRCC of 0.84 is on par with NISQA’s per-utterance performance on clean speech [11], suggesting comparable per-sample reliability.
### VI-E TA Head Performance
The textual alignment (TA) head achieves substantially lower correlation (SRCC $∼$ 0.60) than the musical impression (MI) head (SRCC $∼$ 0.84). This gap likely reflects a fundamental architectural limitation: TA assessment requires comparing audio content against the text prompt, which our architecture cannot access, as it processes only the audio waveform. Text-conditioned architectures (e.g., using CLAP embeddings of the prompt) would be needed to meaningfully address text-audio alignment prediction.
### VI-F Limitations
#### Generalization.
All results are on MusicEval with 31 TTM systems. Cross-dataset generalization (e.g., to SongEval [26]) is untested, and bootstrap CIs on 31 systems are wide (A1 lower bound: 0.898, close to the 0.90 target). Additionally, the MI dimension captures expert musical impression, which may not align with end-user preferences across genres or cultural contexts.
#### Degradation sensitivity.
The metric is insensitive to musical-structural distortions (pitch shift concordance $0.43$ – $0.51$ , tempo stretch $0.46$ – $0.58$ ), with overall concordance ( $0.63$ ) well below the $0.85$ target, limiting its utility for detecting pitch- or tempo-related generation errors (Table V).
#### Single-fold analyses.
Both the data efficiency (Table VII) and degradation experiments use fold 0 only. Reported trends (e.g., the 250-sample crossover) may shift with different fold compositions. The $n=31$ system means in the Steiger test also share audio prompts, which may inflate type I error.
## VII Conclusion
We presented MuQ-Eval, an open-source per-sample quality metric for generated music. Our systematic investigation reveals that frozen representations from a music understanding encoder (MuQ-310M) with a simple MLP head achieve system-level SRCC $=0.957$ with expert quality ratings, comparable to the closed-source DORA-MOS system and $4.8×$ higher than Audiobox Aesthetics. A progressive ablation over training objectives, encoder adaptation, and multi-task strategies shows that additional complexity does not meaningfully improve over this simple baseline, with all component deltas below $Δ=0.01$ at system level. Encoder choice is the single most impactful design decision.
A data efficiency analysis further reveals that LoRA-adapted models are remarkably sample-efficient: only 150 training clips (comparable to a single artist’s discography) yield utterance-level SRCC $=0.761$ , and 250 clips suffice to match the frozen baseline trained on 500. This low annotation barrier makes personalized quality evaluators practical: a single listener can rate one artist’s song catalog and then use the resulting evaluator to score generated music in that style, enabling style-specific quality control without broad annotation campaigns.
A controlled degradation analysis reveals selective sensitivity: the metric reliably detects signal-level artifacts (MP3 compression concordance $=0.97$ , noise $=0.99$ at severe levels) but is insensitive to musical-structural distortions (pitch shift $0.43$ – $0.51$ , tempo stretch $0.46$ – $0.58$ ), consistent with MuQ’s pre-training normalizing pitch and tempo variation.
These results establish that the “deep features + quality annotations” recipe proven in vision (LPIPS) and speech (NISQA, DNSMOS) transfers effectively to music generation evaluation, and that the primary bottleneck is not the training recipe but the encoder’s pre-training quality. The modest data requirement, low computational cost (trainable on a single general-purpose consumer GPU), and demonstrated sample efficiency make this approach accessible to individual researchers without large-scale annotation infrastructure or dedicated compute clusters. MuQ-Eval provides the community with a practical, open tool for per-sample quality assessment of generated music, requiring only 35 ms and 3 GB VRAM per clip.
#### Future work.
Key directions include: (1) cross-dataset validation on SongEval and other benchmarks; (2) improving sensitivity to musical-structural distortions (pitch, tempo) via augmentation-aware training or pitch/tempo-specific auxiliary heads; (3) text-conditioned architectures for textual alignment prediction; (4) evaluation on larger TTM system pools for tighter confidence intervals; (5) investigation of whether fine-tuning benefits emerge with larger quality annotation datasets; and (6) personalized evaluators trained on individual listener preferences, leveraging the low data requirement to build style-specific or genre-specific quality metrics from small personal collections.
## References
- [1] J. Copet et al., “Simple and controllable music generation,” in Proc. NeurIPS, 2023.
- [2] A. Agostinelli et al., “MusicLM: Generating music from text,” arXiv:2301.11325, 2023.
- [3] Z. Evans et al., “Stable Audio Open,” arXiv:2407.14358, 2024.
- [4] K. Kilgour, M. Zuluaga, D. Roblek, and M. Sharifi, “Fréchet Audio Distance: A metric for evaluating music enhancement algorithms,” in Proc. Interspeech, 2019. arXiv:1812.08466.
- [5] Y. Huang et al., “MAD: Aligning text-to-music evaluation with human preferences,” arXiv:2503.16669, 2025.
- [6] Y. Wu et al., “Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,” in Proc. ICASSP, 2023.
- [7] B. Elizalde et al., “CLAP: Learning audio concepts from natural language supervision,” in Proc. ICASSP, 2023. arXiv:2206.04769.
- [8] H. Zhang et al., “From aesthetics to human preferences: Comparative perspectives of evaluating text-to-music systems,” in Proc. IEEE MLSP, 2025. arXiv:2504.21815.
- [9] W.-C. Huang et al., “The AudioMOS Challenge 2025,” arXiv:2509.01336, 2025.
- [10] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, “The unreasonable effectiveness of deep features as a perceptual metric,” in Proc. CVPR, 2018. arXiv:1801.03924.
- [11] G. Mittag, B. Naderi, A. Chehadi, and S. Möller, “NISQA: A deep CNN-self-attention model for multidimensional speech quality prediction with crowdsourced datasets,” in Proc. Interspeech, 2021.
- [12] C. K. A. Reddy et al., “DNSMOS P.835—A non-intrusive perceptual objective speech quality metric to evaluate noise suppressors,” in Proc. ICASSP, 2022.
- [13] C. Liu et al., “MusicEval: A generative music dataset with expert ratings,” arXiv:2501.10811, 2025.
- [14] H. Zhu et al., “MuQ: Self-supervised music representation learning with Mel-residual vector quantization,” arXiv:2501.01108, 2025.
- [15] Y. Li et al., “MERT: Acoustic music understanding model with large-scale self-supervised training,” in Proc. ICLR, 2024. arXiv:2306.00107.
- [16] M. A. Kader et al., “A survey on evaluation metrics for music generation,” arXiv:2509.00051, 2025.
- [17] M. Tailleur et al., “Correlation of Fréchet Audio Distance with human perception is embedding dependent,” in Proc. EUSIPCO, 2024. arXiv:2403.17508.
- [18] A. Gui et al., “Adapting Fréchet Audio Distance for generative music evaluation,” in Proc. ICASSP, 2024. arXiv:2311.01616.
- [19] F. Grotschla et al., “Benchmarking music generation models and metrics via human preference studies,” in Proc. ICASSP, 2025. arXiv:2506.19085.
- [20] Y. Chung et al., “KAD: No more FAD! An effective and efficient evaluation metric for audio generation,” arXiv:2502.15602, 2025.
- [21] T. Takano et al., “Human-CLAP: Human-perception-based contrastive language-audio pretraining,” arXiv:2506.23553, 2025.
- [22] A. Ragano, J. Skoglund, and A. Hines, “SCOREQ: Speech quality assessment with contrastive regression,” in Proc. NeurIPS, 2024. arXiv:2410.06675.
- [23] C. Chen et al., “Audio large language models can be descriptive speech quality evaluators,” in Proc. ICLR, 2025. arXiv:2501.17202.
- [24] S. Deshmukh et al., “PAM: Prompting audio-language models for audio quality assessment,” in Proc. Interspeech, 2024. arXiv:2402.00282.
- [25] A. Tjandra et al., “Meta Audiobox Aesthetics: Unified automatic quality assessment for speech, music, and sound,” arXiv:2502.05139, 2025.
- [26] J. Yao et al., “SongEval: A benchmark dataset for song aesthetics evaluation,” arXiv:2505.10793, 2025.
- [27] E. J. Hu et al., “LoRA: Low-rank adaptation of large language models,” in Proc. ICLR, 2022. arXiv:2106.09685.
- [28] A. Kendall, Y. Gal, and R. Cipolla, “Multi-task learning using uncertainty to weigh losses for scene geometry and semantics,” in Proc. CVPR, 2018.