# SPG-Codec: Exploring the Role and Boundaries of Semantic Priors in Ultra-Low-Bitrate Neural Speech Coding
**Authors**: Mingyu Zhao, Zijian Lin, Kun Wei, Zhiyong Wu
> ∗ These authors contributed equally. † Corresponding author.
## Abstract
Conventional neural speech codecs suffer from severe intelligibility degradation at ultra-low bitrates, where the bottleneck transitions from acoustic distortion to semantic loss. To address this issue, this paper conducts a systematic investigation into the role and fundamental limits of integrating frozen semantic priors – specifically HuBERT and Whisper – into neural speech coding. We introduce and quantitatively validate a novel Semantic Retirement phenomenon: while semantic constraints reduce the Word Error Rate (WER) by up to ${∼}10\$ relatively at 1.5 kbps, their benefits rapidly diminish beyond 6 kbps, indicating a practical capacity boundary. We further uncover a clear trade-off between different prior types: acoustic-rich priors (HuBERT) better preserve prosodic and timbral details, whereas high-level linguistic priors (Whisper) effectively suppress phonetic hallucinations in noisy environments (reducing hallucination rates by 26 percent) and substantially narrow the generalization gap for unseen speakers. Building on these findings, we propose a bitrate-aware regulation strategy that dynamically adjusts prior strength to optimize the trade-off between semantic consistency and perceptual naturalness. Extensive experimental evaluations confirm that our approach achieves competitive intelligibility and noise robustness compared to existing baselines, offering a principled pathway toward ultra-low-bitrate generative speech coding.
## I Introduction
The landscape of speech coding has undergone a paradigm shift with the advent of deep learning. While traditional digital signal processing (DSP) codecs like Opus and EVS have dominated telecommunications for decades, neural audio codecs have recently established new benchmarks for efficiency and quality. Early neural approaches explored WaveNet-based architectures and end-to-end optimization, laying the groundwork for modern systems. By integrating Residual Vector Quantization (RVQ) with Generative Adversarial Networks (GANs) [14], state-of-the-art architectures such as SoundStream [30], EnCodec [6], and HiFi-Codec [28] achieve high-fidelity reconstruction at bitrates as low as 3–6 kbps. Beyond compression, these discrete representations act as the fundamental vocabulary for the rapidly evolving field of Speech Large Language Models (SpeechLLMs) [26, 2, 19], empowering systems like VALL-E [26, 3] and Moshi [7] to generate human-parity speech.
Despite these advancements, a formidable barrier remains in the ultra-low bitrate regime (e.g., $≤$ 1.5 kbps). In such extreme compression scenarios, the available bandwidth is often insufficient to encode both linguistic content and acoustic details (e.g., timbre, prosody) solely from the waveform. Consequently, traditional neural codecs suffer from severe degradation: phonemes collapse or blur, rendering speech intelligible but semantically ambiguous, a phenomenon we refer to as semantic collapse. Even recent large-scale trained models like BigCodec [27] and flow-matching approaches [5] face challenges in balancing bitrate and semantic integrity. This indicates that the coding bottleneck has shifted from minimizing spectral distortion (an acoustic objective) to preserving linguistic integrity (a semantic objective), a challenge recently highlighted by [29]. To overcome this severe information bottleneck, the encoder must leverage external semantic priors to hallucinate or reconstruct plausible speech content from compressed cues.
In response to this challenge, a new wave of semantic-aware codecs has emerged. Innovations like SpeechTokenizer [31] and RepCodec [12] aim to disentangle semantic tokens from acoustic residuals. Similarly, frameworks such as SemantiCodec [17] and SNAC [23] employ multi-scale hierarchies to capture coarse-grained semantic features. Recent works like WavTokenizer [13] and DualCodec [16] further align discrete codes with the semantic space of LLMs. While these methods demonstrate the potential of incorporating semantics, they primarily focus on architectural designs for generative tasks. A critical gap remains in understanding the fundamental mechanisms and limits of semantic guidance: (1) Under what conditions does semantic information become essential versus redundant? (2) How do different types of priors—ranging from acoustic-rich SSL features to abstract ASR transcriptions—impact the trade-off between intelligibility and naturalness?
In this paper, we propose a unified analysis framework to systematically investigate the role of frozen semantic priors in ultra-low-bitrate speech coding. To this end, we augment a standard neural codec with auxiliary semantic conditioning from pre-trained foundation models, specifically HuBERT [11] and Whisper [20]. This setup allows us to quantitatively isolate the benefits of semantic constraints and observe their interaction with bitrate allocation. Our contributions are fourfold:
First, we quantitatively define the Semantic Retirement phenomenon, identifying a 6 kbps boundary where priors shift from essential (reducing WER by ${∼}10\$ relatively at 1.5 kbps) to redundant. Second, we reveal a trade-off between prior types: HuBERT preserves acoustic texture, while Whisper suppresses hallucinations (by 26%) and ensures linguistic consistency. Third, we demonstrate that semantic priors bridge the generalization gap on unseen speakers by over 15% and enhance noise robustness. Finally, we propose a bitrate-aware regulation strategy that dynamically adjusts prior weights to balance semantic integrity and perceptual quality.
## II Related Work
### II-A Neural Audio Codecs
The transition from parametric coding to neural waveform modeling has revolutionized speech compression. Early neural approaches applied VQ-VAE [25] to learn discrete representations from raw audio, though often suffering from spectral smoothing. The introduction of GAN-based discriminators [14] significantly mitigated this by enforcing realistic phase and texture reconstruction. Building on these foundations, end-to-end codecs like SoundStream [30] and EnCodec [6] employed Residual Vector Quantization (RVQ) to achieve scalable bitrates within a single model. Recent advancements continue to refine this paradigm: HiFi-Codec [28] integrates Group-RVQ to reduce quantization errors, while DAC [15] addresses aliasing artifacts for high-fidelity music generation. More recently, large-scale models like BigCodec [27] and TiCodec [21] have pushed the boundaries of acoustic modeling. However, despite their success at medium bitrates (e.g., 6 kbps), purely acoustic-driven models face a theoretical bottleneck at ultra-low bitrates ( $<2$ kbps), where the scarcity of bits leads to the collapse of phonetic structures, necessitating external information injection.
### II-B Semantic-Aware Coding and Speech Generative Models
To overcome the limitations of acoustic-only modeling, recent research has pivoted towards “semantic-aware” coding, largely driven by the needs of SpeechLLMs. One prevalent strategy is disentanglement: SpeechTokenizer [31] and RepCodec [12] decouple speech into semantic tokens (for content) and acoustic tokens (for paralinguistics). Similarly, DualCodec [16] and SACodec [8] explicitly introduce semantic streams to enhance low-bitrate performance. This trend aligns with the rise of generative audio models. Discrete token-based systems like VALL-E 2 [3] and CosyVoice 3 [9] leverage these codes for zero-shot synthesis. Furthermore, the emergence of flow matching models, such as F5-TTS [5] and E2 TTS [10], demonstrates that strong semantic conditioning can drive high-fidelity generation. While these works successfully utilize semantic information for generation or architectural design, they lack a systematic, quantitative analysis of the boundary conditions: specifically, at what exact bitrate does semantic guidance become redundant (“Semantic Retirement”), and how do different priors trade off between intelligibility and acoustic fidelity?
It is worth noting that our analysis framework is complementary rather than competitive to these architectural works: the Semantic Retirement phenomenon and its 6 kbps boundary are diagnostic findings applicable across codec architectures, including SpeechTokenizer, DualCodec, and SemantiCodec.
### II-C Self-Supervised Speech Representations
Self-supervised learning (SSL) serves as the cornerstone for extracting semantic priors. Contrastive frameworks like wav2vec 2.0 [1] and masked prediction models like HuBERT [11] and WavLM [4] capture rich acoustic and linguistic structures. WavLM specifically enhances robustness via denoising tasks, motivating our investigation into noise robustness. Conversely, weakly supervised models trained on massive labeled datasets, most notably Whisper [20], demonstrate exceptional capabilities in ASR, capturing high-level semantics invariant to speaker characteristics. In this paper, rather than training new features, we employ these pre-trained models as frozen priors to regulate the codec. By contrasting acoustic-heavy priors (HuBERT) with semantic-heavy priors (Whisper), we provide a granular analysis of their respective roles in bridging the bit-rate gap.
## III Methodology
In this section, we systematically analyze the interplay between acoustic fidelity and semantic guidance in ultra-low-bitrate coding. To facilitate this analysis, we propose a unified framework (as illustrated in Fig. 1) comprising three core components: a scalable neural codec backbone, a frozen semantic constraint module, and a bitrate-aware regulation strategy. This design aims to quantitatively isolate the contribution of semantic priors and dynamically balance semantic consistency with perceptual quality.
<details>
<summary>2604.26296v1/x1.png Details</summary>
### Visual Description
## Diagram: Semantic Prior-Guided Neural Speech Codec
### Overview
The diagram illustrates a neural speech codec architecture that integrates semantic constraints to improve speech reconstruction. The system processes input audio (X) through an encoder, quantizes latent representations via a rate-variable quantizer (RVQ), and decodes them using a HiFi-GAN. A semantic constraint module, featuring frozen SSL models (HuBERT/Whisper), guides the system to preserve semantic content during compression.
### Components/Axes
1. **Core Components**:
- **Encoder**: CNN-based, processes input X into latent representation `z`.
- **RVQ**: Rate-variable quantizer with 4 bitrate options (1.5/3/6/12 kbps), outputs quantized `ẑ`.
- **Decoder**: HiFi-GAN reconstructs output `X̂` from `ẑ`.
- **Semantic Constraint Module** (frozen):
- **SSL Models**: HuBERT/Whisper for semantic analysis.
- **Loss Functions**:
- `L_sem`: Semantic loss (blue) comparing target `z_tgt` and reconstructed `z_rec`.
- `L_rec`: Reconstruction loss (orange) between `X` and `X̂`.
- **Alpha Adjustment**: Rate-aware scaling factor (α) modulating `L_sem` based on bitrate R.
2. **Flow**:
- Input → Encoder → RVQ → Decoder → Output.
- Semantic module operates in parallel, feeding losses into the total loss function `L = L_rec + α·L_sem`.
### Detailed Analysis
- **Encoder/Decoder**: Standard CNN and HiFi-GAN components for audio compression/reconstruction.
- **RVQ**: Enables variable bitrate compression (1.5–12 kbps), critical for bandwidth adaptation.
- **Semantic Module**:
- **HuBERT/Whisper**: Pre-trained SSL models (frozen) extract semantic features from `z` and `z_rec`.
- **Losses**:
- `L_sem`: Measures semantic fidelity between target and reconstructed latent codes.
- `L_rec`: Standard reconstruction loss (MSE/PESQ-like metric).
- **Alpha Adjustment**: Dynamically scales `L_sem` based on bitrate R, balancing quality and compression efficiency.
### Key Observations
1. **Trade-off Mechanism**: Higher bitrates (e.g., 12 kbps) likely reduce α, prioritizing reconstruction over semantic constraints.
2. **Frozen SSL Models**: Indicates pre-trained models are used for semantic guidance without fine-tuning during codec training.
3. **Shared Weights**: `z_tgt` and `z_rec` share weights, suggesting a unified semantic representation space.
### Interpretation
The architecture demonstrates a hybrid approach to speech coding, combining traditional compression (via RVQ/HiFi-GAN) with modern SSL-based semantic preservation. The rate-aware α adjustment allows adaptive optimization: at low bitrates, semantic fidelity is emphasized to maintain intelligibility, while higher bitrates prioritize raw reconstruction quality. The frozen SSL models suggest a plug-and-play integration of semantic constraints, avoiding the computational overhead of joint training. This design could enable efficient, high-quality speech transmission in bandwidth-constrained scenarios (e.g., VoIP, IoT devices) while maintaining naturalness through semantic guidance.
</details>
Figure 1: Framework of our proposed SPG-Codec. A frozen Semantic Constraint Module (HuBERT/Whisper) guides the backbone, while the $α$ -Adjust module dynamically modulates semantic weight based on bitrate to address “Semantic Retirement”. Shared weights: a single frozen encoder instance is applied twice in forward-pass mode—once to $x$ to produce $z^Φ_tgt$ and once to $\hat{x}$ to produce $z^Φ_rec$ —with no additional learnable parameters.
### III-A Backbone Neural Codec
We adopt a convolutional encoder-decoder architecture akin to SoundStream [30] and EnCodec [6] as the backbone. Let $x∈ℝ^T$ be the input waveform. The encoder $E$ downsamples $x$ into a latent representation $z=E(x)∈ℝ^D× T^{\prime}$ , capturing local acoustic features.
Scalable Quantization. To support variable bitrates (1.5–12 kbps) within a single model, we employ Residual Vector Quantization (RVQ). The quantizer $Q$ approximates $z$ as a sum of discrete vectors:
$$
\hat{z}=∑_k=1^N_q\hat{z}_k, where \hat{z}_k=\mathop{\arg\min}_c∈C_k\|r_k-1-c\|_2 \tag{1}
$$
Here, $r_0=z$ , $r_k=r_k-1-\hat{z}_k$ is the residual, and $C_k$ is the codebook for the $k$ -th layer. The number of active quantizers $N_q$ is dynamically adjusted during inference to determine the bitrate $R$ .
Generator & Discriminator. The decoder $G$ reconstructs the waveform $\hat{x}=G(\hat{z})$ using a HiFi-GAN-based architecture [14]. To ensure perceptual quality, the training objective includes a multi-resolution spectral loss $L_spec$ and an adversarial loss $L_adv$ derived from a Multi-Period Discriminator (MPD) and a Multi-Scale Discriminator (MSD).
### III-B Semantic Constraint Module
Standard acoustic losses (e.g., $L_1$ and Spectral) penalize waveform differences but often fail to preserve phonemic integrity at ultra-low bitrates ( $≤ 1.5$ kbps), leading to “mumbled” speech. To mitigate this, we introduce a Semantic Constraint Module that injects linguistic priors into the reconstruction process. We investigate two distinct paradigms:
1) Acoustic-Semantic Prior (HuBERT [11]): HuBERT is trained via masked prediction of hidden units. It effectively captures both phonetic content and paralinguistic features (e.g., prosody, speaker identity). We extract representations from the intermediate transformer layers, which are known to encode rich acoustic-phonetic information:
$$
z^H_tgt=Φ_HuBERT(x), z^H_rec=Φ_HuBERT(\hat{x}) \tag{2}
$$
Here, $\hat{x}$ is resampled if necessary to match the input sampling rate of the prior model (16 kHz).
2) Linguistic-Semantic Prior (Whisper [20]): Trained on 680k hours of weak supervision, Whisper’s encoder representations are highly invariant to noise and speaker characteristics, focusing purely on linguistic content.
$$
z^W_tgt=Φ_Whisper(x), z^W_rec=Φ_Whisper(\hat{x}) \tag{3}
$$
We hypothesize that Whisper serves as a “loose” constraint, tolerating acoustic mismatch (e.g., slight pitch shift) while strictly enforcing semantic correctness, which is critical for suppressing hallucinations in noisy conditions.
The semantic loss is formulated as the $L_1$ distance between layer-normalized feature maps:
$$
L_sem(x,\hat{x})=\|LN(Φ(x))-LN(Φ(\hat{x}))\|_1 \tag{4}
$$
where $LN(·)$ denotes Layer Normalization, ensuring numerical stability during training.
### III-C Bitrate-Aware Regulation Strategy
A static weight for semantic loss is suboptimal. Through preliminary experiments, we identified a critical trade-off mechanism, which we term the “Semantic Retirement” phenomenon:
- Low Bitrate (Bottleneck Regime): When $R≤ 3$ kbps, the acoustic information is insufficient to resolve phonemes. The model is prone to “semantic collapse.” Here, a strong semantic prior acts as a teacher, guiding the decoder to hallucinate plausible phonemes from compressed cues.
- High Bitrate (Sufficient Regime): When $R≥ 6$ kbps, the latent code $\hat{z}$ contains sufficient information. Forcing the output to match a pre-trained semantic space—which inherently abstracts away fine-grained acoustic details—creates a gradient conflict with the reconstruction loss ( $L_rec$ ). This leads to over-smoothing and degraded perceptual naturalness (e.g., lower PESQ), as observed in recent generative codecs [9].
To resolve this conflict, we propose a Bitrate-Aware Regulation mechanism. We define a dynamic weight $α(R)$ inversely correlated with bitrate $R$ . The total training objective is:
$$
L_total=L_rec+λ_advL_adv+α(R)·L_sem \tag{5}
$$
where $L_rec=L_L1+L_spec$ comprises the $\ell_1$ waveform reconstruction loss and the multi-resolution spectral loss defined in Sec. III-A. Based on our empirical analysis (detailed in Sec. IV), we adopt a step-decay strategy: $α(R)$ is maximized (e.g., $α=0.1$ ) in the bottleneck regime to prioritize intelligibility, and decays to a negligible value (e.g., $α=0.01$ ) in the sufficient regime to prioritize fidelity. This strategy ensures the model traverses the Pareto frontier of Intelligibility (WER) vs. Quality (PESQ).
## IV Experiments
### IV-A Experimental Setup
Dataset and Implementation. We evaluate our framework on the LibriSpeech dataset [18]. All models are trained on the train-clean-100 split and evaluated on the test-clean and test-other sets. The audio is sampled at 16 kHz. Our backbone codec adopts a SoundStream-like architecture with a causal convolutional encoder and decoder. The Residual Vector Quantizer (RVQ) is configured to support scalable bitrates of 1.5, 3.0, 6.0, and 12.0 kbps. For semantic priors, we utilize the pre-trained HuBERT-base [11] and Whisper-base [20] models. The encoders of these models are frozen during training. The dynamic weight $α$ is set to 0.1 for low bitrates based on preliminary sweeps, decaying to 0.01 for high bitrates ( $R≥ 6$ kbps). All models are trained on 8 NVIDIA V100 GPUs for 100 epochs with a batch size of 64.
Evaluation Metrics. We assess performance using both objective quality and semantic consistency metrics:
- Perceptual Quality: We report PESQ (Wideband) [22], STOI [24], SI-SDR, and the $\ell_1$ waveform reconstruction loss (used as an internal diagnostic metric).
- Semantic Consistency: We measure Word Error Rate (WER) using a separate pre-trained ASR model (Whisper-large-v2) to evaluate intelligibility.
- Speaker Similarity: We compute the cosine similarity between WavLM embeddings of the ground truth and reconstructed speech to assess speaker identity preservation.
### IV-B The Phenomenon of Semantic Retirement
We first validate the core hypothesis of “Semantic Retirement.” Fig. 2 illustrates the relative performance improvement of introducing semantic priors over the baseline across different bitrates.
<details>
<summary>2604.26296v1/x2.png Details</summary>
### Visual Description
## Line/Bar Chart: WER and PESQ Improvement Comparison Across Bitrates
### Overview
The chart compares Word Error Rate (WER) and Perceptual Evaluation of Speech Quality (PESQ) improvements between two models (HuBERT and Whisper) across four bitrates (1.5, 3.0, 6.0, and 12.0 kbp). It includes two y-axes: left for WER improvement (%) and right for PESQ improvement (%), with a legend distinguishing the models by color (blue for HuBERT, red for Whisper). Two zones are annotated: "Prior Effective Zone" (green shading) and "Diminishing Returns" (gray shading), with a "Semantic Retirement Boundary" line (green dashed) connecting PESQ data points at 6.0 and 12.0 kbp.
### Components/Axes
- **X-axis**: Bitrate (kbp), labeled with markers at 1.5, 3.0, 6.0, and 12.0.
- **Left Y-axis**: Relative WER Improvement (%), ranging from -5% to 35%.
- **Right Y-axis**: PESQ Improvement (%), ranging from -3% to 3%.
- **Legend**: Top-left, with blue squares for HuBERT and red squares for Whisper.
- **Zones**:
- "Prior Effective Zone" (green shading, left of dashed vertical line at 3.0 kbp).
- "Diminishing Returns" (gray shading, right of dashed line).
- **Semantic Retirement Boundary**: Green dashed line connecting PESQ data points at 6.0 and 12.0 kbp.
### Detailed Analysis
#### WER Improvement (Left Y-axis)
- **HuBERT (blue bars)**:
- 1.5 kbp: ~27% improvement.
- 3.0 kbp: ~16% improvement.
- 6.0 kbp: ~0% improvement.
- 12.0 kbp: ~0% improvement.
- **Whisper (red bars)**:
- 1.5 kbp: ~15% improvement.
- 3.0 kbp: ~12% improvement.
- 6.0 kbp: ~0% improvement.
- 12.0 kbp: ~0% improvement.
#### PESQ Improvement (Right Y-axis)
- **HuBERT (blue line)**:
- 1.5 kbp: ~23% improvement.
- 3.0 kbp: ~21% improvement.
- 6.0 kbp: ~13% improvement.
- 12.0 kbp: ~25% improvement.
- **Whisper (red line)**:
- 1.5 kbp: ~22% improvement.
- 3.0 kbp: ~23% improvement.
- 6.0 kbp: ~3% improvement.
- 12.0 kbp: ~18% improvement.
### Key Observations
1. **WER Trends**:
- Both models show significant WER improvement at 1.5–3.0 kbp, with HuBERT outperforming Whisper.
- WER improvement plateaus near 0% at 6.0 and 12.0 kbp, indicating no further gains beyond 3.0 kbp.
2. **PESQ Trends**:
- HuBERT maintains higher PESQ improvement across all bitrates, with a notable rebound at 12.0 kbp.
- Whisper’s PESQ improvement drops sharply at 6.0 kbp and recovers slightly at 12.0 kbp.
3. **Zones**:
- The "Prior Effective Zone" (1.5–3.0 kbp) aligns with peak performance for both metrics.
- The "Diminishing Returns" zone (6.0–12.0 kbp) shows reduced gains, particularly for Whisper.
### Interpretation
The data suggests that **lower bitrates (1.5–3.0 kbp)** are optimal for both WER and PESQ improvements, with HuBERT consistently outperforming Whisper. Beyond 3.0 kbp, WER improvements stagnate, while PESQ improvements for Whisper decline significantly, highlighting a trade-off between bitrate and quality. The "Semantic Retirement Boundary" implies that higher bitrates (6.0–12.0 kbp) may not yield meaningful semantic gains, especially for Whisper. HuBERT’s resilience at 12.0 kbp suggests better scalability, but both models exhibit diminishing returns in the gray-shaded zone. This aligns with the hypothesis that compression beyond 3.0 kbp offers limited practical benefits for speech quality and error reduction.
</details>
Figure 2: The “Semantic Retirement” Phenomenon. Relative improvement (%) over the no-prior baseline of semantic priors (HuBERT/Whisper). Gains are substantial at 1.5 kbps but diminish above 6.0 kbps.
Low-Bitrate Regime (1.5–3.0 kbps). As shown in the “Prior Effective Zone” of Fig. 2, semantic priors provide substantial gains. At 1.5 kbps, the HuBERT prior improves the $L_1$ loss by 27.1% and reduces WER by ${∼}10\$ , confirming that semantic knowledge compensates for the missing bits. High-Bitrate Regime ( $≥$ 6.0 kbps). As the bitrate increases, the benefit of semantic priors diminishes rapidly. At 6.0 kbps, the relative improvement in PESQ drops to near zero (or slightly negative for Whisper), and the WER improvement becomes negligible ( $<1\$ ). This validates the “Semantic Retirement” boundary: once the codebook capacity is sufficient to encode acoustic details, enforcing a semantic constraint becomes redundant and may even conflict with the reconstruction objective.
### IV-C Trade-off Analysis: Acoustic vs. Semantic Priors
Different priors exhibit distinct characteristics. Fig. 3 presents a comprehensive comparison at 3.0 kbps.
<details>
<summary>2604.26296v1/x3.png Details</summary>
### Visual Description
## Radar Chart: Prior Trade-off @ 3.0 kbps
### Overview
The image is a radar chart comparing three data series ("Baseline," "+ HuBERT," and "+ Whisper") across four metrics: Semantic Consistency, Speaker Similarity, PESQ, and Mel Accuracy. The chart uses a circular layout with axes radiating from a central point, and data points are plotted as markers connected by lines. The legend is positioned in the top-right corner, and the chart title is centered at the top.
---
### Components/Axes
- **Axes (Metrics)**:
- **Semantic Consistency** (top-right, 0.75–1.0 scale)
- **Speaker Similarity** (top-left, 0.75–1.0 scale)
- **PESQ** (bottom-left, 0.75–1.0 scale)
- **Mel Accuracy** (bottom-right, 0.75–1.0 scale)
- **Legend**:
- **Baseline**: Gray dashed line with circle markers
- **+ HuBERT**: Blue solid line with square markers
- **+ Whisper**: Red solid line with star markers
- **Axes Markers**: Concentric circles with labels at 0.75, 0.80, 0.85, 0.90, 0.95, and 1.0.
---
### Detailed Analysis
1. **Baseline (Gray)**:
- **Semantic Consistency**: ~0.85
- **Speaker Similarity**: ~0.80
- **PESQ**: ~0.75
- **Mel Accuracy**: ~0.75
- **Trend**: Lowest values across all metrics, forming a compact polygon near the center.
2. **+ HuBERT (Blue)**:
- **Semantic Consistency**: ~0.95
- **Speaker Similarity**: ~0.95
- **PESQ**: ~0.90
- **Mel Accuracy**: ~0.90
- **Trend**: High values in Semantic Consistency and Speaker Similarity, with moderate improvements in PESQ and Mel Accuracy compared to Baseline.
3. **+ Whisper (Red)**:
- **Semantic Consistency**: ~0.90
- **Speaker Similarity**: ~0.95
- **PESQ**: ~0.90
- **Mel Accuracy**: ~0.90
- **Trend**: Peaks at Speaker Similarity (~0.95), with slightly lower but still strong values in other metrics.
---
### Key Observations
- **Whisper** achieves the highest value in **Speaker Similarity** (~0.95) but lags slightly behind HuBERT in Semantic Consistency (~0.90 vs. 0.95).
- **HuBERT** dominates in **Semantic Consistency** and **Speaker Similarity**, with balanced performance in PESQ and Mel Accuracy.
- **Baseline** underperforms across all metrics, with the lowest values (~0.75–0.85).
- **+ HuBERT** and **+ Whisper** show similar trade-offs: strong gains in Semantic Consistency/Speaker Similarity at the cost of slightly lower PESQ/Mel Accuracy compared to Baseline.
---
### Interpretation
The chart illustrates a trade-off between audio quality metrics (PESQ, Mel Accuracy) and perceptual metrics (Semantic Consistency, Speaker Similarity).
- **HuBERT** prioritizes perceptual quality, excelling in Semantic Consistency and Speaker Similarity but sacrificing some objective quality (PESQ/Mel Accuracy).
- **Whisper** balances perceptual and objective quality, with the highest Speaker Similarity but slightly lower Semantic Consistency than HuBERT.
- **Baseline** represents the lowest performance, serving as a reference for degradation.
The data suggests that HuBERT and Whisper improve perceptual aspects of audio processing at the expense of objective quality metrics, highlighting a common trade-off in audio enhancement systems. The proximity of HuBERT and Whisper in PESQ/Mel Accuracy indicates that perceptual gains may not always correlate with objective quality improvements.
</details>
Figure 3: Trade-off Analysis at 3.0 kbps. Absolute scores on multiple metrics. HuBERT excels in acoustic fidelity (PESQ), while Whisper dominates in Semantic Consistency (lowest absolute WER).
HuBERT (Acoustic-Dominant). HuBERT achieves higher scores in Mel-spectrogram accuracy and Pitch Correlation. This is attributed to its masked prediction objective, which forces the model to retain acoustic features such as prosody and timbre. It serves as a robust prior for preserving the “texture” of speech.
Whisper (Semantic-Dominant). Conversely, Whisper yields the best Semantic Consistency (lowest WER). Since Whisper is trained on large-scale weak supervision for ASR, its representations are highly invariant to speaker and noise but rich in linguistic content. This makes Whisper the ideal choice for maximizing intelligibility in extreme compression scenarios, albeit with a slight trade-off in acoustic detail preservation.
### IV-D Robustness Against Noise and Hallucinations
A critical advantage of semantic priors is their robustness in adverse environments. We evaluated the models under varying Signal-to-Noise Ratios (SNR) ranging from Clean to 0 dB.
<details>
<summary>2604.26296v1/x4.png Details</summary>
### Visual Description
## Line Graph: Noise Robustness @ 1.5 kbps
### Overview
The graph illustrates the relative Word Error Rate (WER) increase under varying noise levels (Signal-to-Noise Ratio, SNR) for three systems: Baseline, Baseline + HuBERT, and Baseline + Whisper. The y-axis represents the percentage increase in WER, while the x-axis shows SNR levels from "Clean" (no noise) to 0dB (highest noise). A shaded area highlights the "Robustness Gain" between Baseline and Whisper.
### Components/Axes
- **X-axis (Noise Level/SNR)**: Labeled "Noise Level (SNR)" with categories: Clean, 20dB, 10dB, 5dB, 0dB.
- **Y-axis (Relative WER Increase)**: Labeled "Relative WER Increase (%)" with a scale from 0 to 60.
- **Legend**: Located in the bottom-right corner, with three entries:
- **Baseline**: Gray circles (○).
- **+ HuBERT**: Blue squares (■).
- **+ Whisper**: Red triangles (▲).
- **Shaded Area**: A pink region between Baseline and Whisper lines, labeled "Robustness Gain" with an arrow pointing to the 5dB mark.
### Detailed Analysis
1. **Baseline (Gray ○)**:
- Clean: 0% WER increase.
- 20dB: ~15% increase.
- 10dB: ~37% increase.
- 5dB: ~49% increase.
- 0dB: ~50% increase.
- **Trend**: Steep upward slope as SNR decreases.
2. **+ HuBERT (Blue ■)**:
- Clean: 0% WER increase.
- 20dB: ~12% increase.
- 10dB: ~28% increase.
- 5dB: ~40% increase.
- 0dB: ~48% increase.
- **Trend**: Less steep than Baseline, showing moderate improvement.
3. **+ Whisper (Red ▲)**:
- Clean: 0% WER increase.
- 20dB: ~11% increase.
- 10dB: ~29% increase.
- 5dB: ~40% increase.
- 0dB: ~48% increase.
- **Trend**: Similar to HuBERT but slightly lower WER increases at 20dB and 10dB.
4. **Shaded Area (Robustness Gain)**:
- Visible between Baseline and Whisper lines, indicating Whisper reduces WER compared to Baseline.
- Largest gain observed at 5dB (Baseline: 49% vs. Whisper: 40%).
### Key Observations
- **Noise Sensitivity**: All systems show increased WER as SNR decreases, with Baseline being the most affected.
- **Robustness Gain**: Whisper consistently outperforms Baseline, with the largest relative improvement at lower SNR levels (e.g., 5dB).
- **HuBERT vs. Whisper**: HuBERT and Whisper perform similarly at 5dB and 0dB but Whisper has a slight edge at 20dB and 10dB.
### Interpretation
The graph demonstrates that adding Whisper to the Baseline system significantly improves noise robustness, reducing WER increases by up to 9% at 5dB SNR. HuBERT also enhances robustness but to a lesser extent. The shaded "Robustness Gain" area visually emphasizes Whisper's effectiveness in noisy conditions, suggesting it is a critical component for maintaining accuracy in low-SNR environments. The convergence of HuBERT and Whisper lines at 0dB implies diminishing returns at extreme noise levels, where both struggle to mitigate errors.
</details>
Figure 4: Noise Robustness. Semantic priors (especially Whisper) suppress WER increase under noisy conditions (shaded area).
As shown in Fig. 4, the baseline model’s performance degrades catastrophically as noise increases. At SNR 5dB, the WER increases by nearly 50% relative to the clean condition. In contrast, the Whisper prior acts as a linguistic denoiser, maintaining significantly lower WER growth (“Robustness Gain”).
<details>
<summary>2604.26296v1/x5.png Details</summary>
### Visual Description
## Bar Chart: Hallucination Suppression @ 1.5 kbps
### Overview
The chart compares hallucination suppression rates across four noise conditions (Clean, SNR 10dB, SNR 5dB, SNR 0dB) using three methods: Baseline, +HuBERT, and +Whisper. Hallucination rates are measured as percentages on the y-axis (20–100%), with suppression improvements annotated as percentage changes.
### Components/Axes
- **X-axis (Noise Condition)**:
Categories: Clean, SNR 10dB, SNR 5dB, SNR 0dB.
- **Y-axis (Hallucination Rate %)**:
Scale: 20% to 100% in 10% increments.
- **Legend**:
- Gray: Baseline
- Blue: +HuBERT
- Red: +Whisper
Positioned at the bottom-right corner.
### Detailed Analysis
1. **Clean Condition**:
- Baseline: ~42% hallucination rate.
- +HuBERT: ~34% (↓26% vs. Baseline).
- +Whisper: ~31% (↓26% vs. Baseline).
2. **SNR 10dB**:
- Baseline: ~87% hallucination rate.
- +HuBERT: ~86% (↓1% vs. Baseline).
- +Whisper: ~80% (↓8% vs. Baseline).
3. **SNR 5dB**:
- Baseline: ~95% hallucination rate.
- +HuBERT: ~91% (↓4% vs. Baseline).
- +Whisper: ~93% (↓2% vs. Baseline).
4. **SNR 0dB**:
- Baseline: ~93% hallucination rate.
- +HuBERT: ~94% (↓1% vs. Baseline).
- +Whisper: ~90% (↓4% vs. Baseline).
### Key Observations
- **Suppression Effectiveness**:
- HuBERT and Whisper reduce hallucination rates most significantly in the Clean condition (↓26%).
- In SNR 10dB, Whisper achieves the largest suppression (↓8%), while HuBERT shows minimal improvement.
- At SNR 0dB, Baseline and HuBERT perform similarly, but Whisper lags by 4%.
- **Anomalies**:
- Whisper underperforms Baseline in SNR 0dB despite prior improvements.
- SNR 5dB shows the smallest suppression gains across all methods.
### Interpretation
The data demonstrates that HuBERT and Whisper consistently reduce hallucination rates, with the most dramatic improvements in low-noise (Clean) and moderate-noise (SNR 10dB) conditions. However, performance degrades in high-noise (SNR 0dB) scenarios, particularly for Whisper. This suggests that noise robustness varies between methods, with HuBERT maintaining more consistent suppression across SNR levels. The anomalies in SNR 0dB and 5dB may indicate trade-offs between noise tolerance and model complexity. For applications requiring reliability in noisy environments, HuBERT appears more stable, while Whisper excels in cleaner conditions.
</details>
Figure 5: Hallucination Rate Analysis. Absolute hallucination rates under a separate detection protocol. Whisper effectively suppresses phonetic hallucinations, reducing the absolute rate by 26% in clean conditions at 1.5 kbps.
Furthermore, Fig. 5 highlights the suppression of hallucinations —a common failure mode where the codec generates plausible but incorrect phonemes due to ambiguity. At 1.5 kbps, Whisper reduces the hallucination rate by roughly 26% in clean conditions, ensuring that the reconstructed speech remains faithful to the source text even when acoustic cues are ambiguous.
### IV-E Generalization to Unseen Speakers
To assess whether our method merely overfits to the training distribution or learns generalizable features, we evaluated performance on the challenging LibriSpeech test-other set. As shown in Table I, the baseline model suffers a massive performance drop (WER Gap +36.0%) when shifting from test-clean to test-other. In contrast, introducing the Whisper prior significantly bridges this generalization gap, reducing the degradation to +19.7%. This indicates that high-level semantic priors help the codec maintain intelligibility even for unseen speakers and recording conditions.
TABLE I: Generalization Gap (WER increase from test-clean to test-other) at 1.5 kbps. Lower gap indicates better generalization.
| Baseline + HuBERT + Whisper | 43.4% 38.6% 44.7% | 79.3% 73.7% 64.5% | +35.9% +35.1% +19.7% |
| --- | --- | --- | --- |
### IV-F Ablation Study: Prior Effectiveness & Regulation
Negative Control (Shuffled Prior). To verify that the gains stem from temporal semantic structure rather than mere regularization, we conducted a negative control experiment using shuffled semantic features. At 1.5 kbps, while the normal Whisper prior achieves a high cosine similarity of 0.968 with the target, the shuffled prior drops to 0.374, resulting in a performance gap of $>0.59$ . This confirms that preserving the correct temporal semantic sequence is crucial.
<details>
<summary>2604.26296v1/x6.png Details</summary>
### Visual Description
## Line/Bar Chart: Prior Weight Sensitivity @ 3.0 kbps (HuBERT)
### Overview
The chart visualizes the relationship between prior weight (α) and two metrics: Word Error Rate (WER, %) and Perceptual Evaluation of Speech Quality (PESQ). It combines a bar chart (WER) and line graph (PESQ) with annotations for "Sweet Spot" and "Over-constrained" regions.
### Components/Axes
- **X-axis**: Prior Weight (α) with values: 0.01, 0.05, 0.1, 0.2, 0.5
- **Left Y-axis**: WER (%) ranging from 23.50 to 25.50
- **Right Y-axis**: PESQ ranging from 1800 to 2000
- **Legend**:
- Blue bars: WER (%)
- Red line with circles: PESQ
- **Annotations**:
- Green shaded area labeled "Sweet Spot" (α=0.05–0.1)
- Red arrow labeled "Over-constrained" pointing to α=0.2
### Detailed Analysis
1. **WER (%) (Blue Bars)**:
- α=0.01: ~24.25%
- α=0.05: ~24.00%
- α=0.1: ~24.50%
- α=0.2: ~24.00%
- α=0.5: ~24.50%
- *Trend*: WER decreases slightly from α=0.01 to 0.05, then increases to α=0.1, decreases again at α=0.2, and rises at α=0.5.
2. **PESQ (Red Line)**:
- α=0.01: ~1975
- α=0.05: ~1950
- α=0.1: ~1950
- α=0.2: ~1850 (sharp drop)
- α=0.5: ~1925
- *Trend*: PESQ declines gradually until α=0.1, then plummets at α=0.2, recovering slightly at α=0.5.
3. **Annotations**:
- "Sweet Spot" (green) spans α=0.05–0.1, where WER is lowest and PESQ is stable.
- "Over-constrained" (red arrow) highlights α=0.2, where PESQ drops sharply despite WER remaining low.
### Key Observations
- **Sweet Spot**: Optimal performance occurs at α=0.05–0.1, balancing low WER and stable PESQ.
- **Over-constraint at α=0.2**: PESQ drops ~100 points despite WER remaining near 24%, suggesting a critical threshold where increased prior weight degrades speech quality.
- **U-shaped WER**: WER peaks at α=0.1 and α=0.5, indicating sensitivity to extreme prior weights.
### Interpretation
The data demonstrates a trade-off between WER and PESQ as prior weight (α) increases. While WER improves slightly with higher α (up to α=0.1), PESQ remains stable in the "Sweet Spot." However, at α=0.2, PESQ collapses sharply, suggesting over-constraint harms perceptual quality despite marginal WER improvements. This implies that maximizing WER alone is insufficient; perceptual metrics must guide prior weight selection. The U-shaped WER curve further highlights the need for careful tuning to avoid overfitting or underfitting. The "Over-constrained" annotation underscores a critical failure point where model behavior degrades unexpectedly.
</details>
Figure 6: Impact of Regulation Weight $α$ . A “Sweet Spot” is observed around $α=0.05-0.1$ . Excessive weight ( $α=0.2$ ) leads to an over-constrained system.
Bitrate-Aware Regulation. Finally, we analyze the sensitivity of the semantic weight $α$ to justify our proposed regulation strategy. Fig. 6 plots the WER and PESQ at 3.0 kbps for varying $α$ values. The results reveal a clear “Sweet Spot” around $α∈[0.05,0.1]$ . An overly strong constraint ( $α=0.2$ ) leads to an over-constrained state where PESQ drops sharply (from $≈ 1.96$ to $≈ 1.85$ ), as the model sacrifices acoustic naturalness to satisfy the rigid semantic loss. Consequently, we fix $α=0.1$ for low bitrates and attenuate it to $0.01$ for high bitrates in our final model.
## V Conclusion
We propose a quantitative framework revealing the “Semantic Retirement” phenomenon: frozen priors are indispensable at 1.5 kbps ( ${∼}10\$ WER reduction) but redundant beyond 6 kbps. Acoustic-rich priors (HuBERT) excel in prosody preservation, while linguistic priors (Whisper) enhance noise robustness and suppress hallucinations. Our bitrate-aware regulation dynamically resolves gradient conflicts, advocating a paradigm shift toward semantically-conditioned ultra-low-bitrate generation. Future work will explore end-to-end optimization with large-scale flow matching models.
## VI Acknowledgement
This work is supported by National Natural Science Foundation of China (62076144) and Shenzhen Science and Technology Program (JCYJ20220818101014030).
## References
- [1] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli (2020) wav2vec 2.0: a framework for self-supervised learning of speech representations. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 33, pp. 12449–12460. Cited by: § II-C.
- [2] Z. Borsos, R. Marinier, D. Vincent, E. Kharitonov, O. Pietquin, M. Sharifi, D. Roblek, O. Teboul, D. Grangier, M. Tagliasacchi, and N. Zeghidour (2023) AudioLM: a language modeling approach to audio generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing 31, pp. 2523–2533. Cited by: §I.
- [3] S. Chen, S. Liu, L. Zhou, Y. Liu, X. Tan, J. Li, S. Zhao, Y. Qian, and F. Wei (2024) VALL-E 2: neural codec language models are human parity zero-shot text to speech synthesizers. arXiv preprint arXiv:2406.05370. Cited by: §I, § II-B.
- [4] S. Chen, C. Wang, Z. Chen, Y. Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao, J. Wu, L. Zhou, S. Ren, Y. Qian, Y. Qian, J. Wu, M. Zeng, and F. Wei (2022) WavLM: large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing 16 (6), pp. 1505–1518. External Links: Document Cited by: § II-C.
- [5] Y. Chen, Z. Niu, Z. Ma, K. Deng, C. Wang, J. Zhao, K. Yu, and X. Chen (2025) F5-TTS: a fairytaler that fakes fluent and faithful speech with flow matching. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL), Vienna, Austria, pp. 6255–6271. External Links: Document Cited by: §I, § II-B.
- [6] A. Défossez, J. Copet, G. Synnaeve, and Y. Adi (2023) High fidelity neural audio compression. Transactions on Machine Learning Research (TMLR). External Links: Link Cited by: §I, § II-A, § III-A.
- [7] A. Défossez, L. Mazaré, M. Orsini, A. Royer, P. Pérez, H. Jégou, E. Grave, and N. Zeghidour (2024) Moshi: a speech-text foundation model for real-time dialogue. arXiv preprint arXiv:2410.00037. Cited by: §I.
- [8] Z. Dong, B. Wang, J. Han, H. Guo, X. Mo, Y. Cao, and Z. Zhang (2025) SACodec: asymmetric quantization with semantic anchoring for low-bitrate high-fidelity neural speech codecs. arXiv preprint arXiv:2512.20944. Cited by: § II-B.
- [9] Z. Du, C. Gao, Y. Wang, F. Yu, T. Zhao, H. Wang, X. Lv, H. Wang, X. Shi, K. An, Z. Sheng, Y. Gu, C. Deng, W. Wang, S. Zhang, Z. Yan, and J. Zhou (2025) CosyVoice 3: towards in-the-wild speech generation via scaling-up and post-training. arXiv preprint arXiv:2505.17589. Cited by: § II-B, 2nd item.
- [10] S. E. Eskimez, X. Wang, M. Thakker, C. Li, C. Tsai, Z. Xiao, H. Yang, Z. Zhu, M. Tang, X. Tan, Y. Liu, S. Zhao, and N. Kanda (2024) E2 TTS: embarrassingly easy fully non-autoregressive zero-shot tts. arXiv preprint arXiv:2406.18009. Cited by: § II-B.
- [11] W. Hsu, B. Bolte, Y. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed (2021) HuBERT: self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing 29, pp. 3451–3460. External Links: Document Cited by: §I, § II-C, § III-B, § IV-A.
- [12] Z. Huang, C. Meng, and T. Ko (2024) RepCodec: a speech representation codec for speech tokenization. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), Bangkok, Thailand, pp. 5777–5790. Cited by: §I, § II-B.
- [13] S. Ji, Z. Jiang, W. Wang, Y. Chen, M. Fang, J. Zuo, Q. Yang, X. Cheng, Z. Wang, R. Li, Z. Zhang, X. Yang, R. Huang, Y. Jiang, Q. Chen, S. Zheng, and Z. Zhao (2024) WavTokenizer: an efficient acoustic discrete codec tokenizer for audio language modeling. arXiv preprint arXiv:2408.16532. Note: Accepted to ICLR 2025 Cited by: §I.
- [14] J. Kong, J. Kim, and J. Bae (2020) HiFi-GAN: generative adversarial networks for efficient and high fidelity speech synthesis. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 33, pp. 17022–17033. Cited by: §I, § II-A, § III-A.
- [15] R. Kumar, P. Seetharaman, A. Luebs, I. Kumar, and K. Kumar (2023) High-fidelity audio compression with improved RVQGAN. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 36. Cited by: § II-A.
- [16] J. Li, X. Lin, Z. Li, S. Huang, Y. Wang, C. Wang, Z. Zhan, and Z. Wu (2025) DualCodec: a low-frame-rate, semantically-enhanced neural audio codec for speech generation. In Proceedings of Interspeech, Note: arXiv:2505.13000 Cited by: §I, § II-B.
- [17] H. Liu, X. Xu, Y. Yuan, M. Wu, W. Wang, and M. D. Plumbley (2024) SemantiCodec: an ultra low bitrate semantic audio codec for general sound. IEEE Journal of Selected Topics in Signal Processing 18 (8), pp. 1448–1461. External Links: Document Cited by: §I.
- [18] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur (2015) LibriSpeech: an ASR corpus based on public domain audio books. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5206–5210. Cited by: § IV-A.
- [19] J. Peng, Y. Wang, B. Li, Y. Guo, H. Wang, Y. Fang, Y. Xi, H. Li, X. Li, K. Zhang, S. Wang, and K. Yu (2025) A survey on speech large language models for understanding. arXiv preprint arXiv:2410.18908. Cited by: §I.
- [20] A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever (2023) Robust speech recognition via large-scale weak supervision. In International Conference on Machine Learning (ICML), pp. 28492–28518. Cited by: §I, § II-C, § III-B, § IV-A.
- [21] Y. Ren, T. Wang, J. Yi, L. Xu, J. Tao, C. Y. Zhang, and J. Zhou (2024) Fewer-token neural speech codec with time-invariant codes. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 12737–12741. Cited by: § II-A.
- [22] A. W. Rix, J. G. Beerends, M. P. Hollier, and A. P. Hekstra (2001) Perceptual evaluation of speech quality (PESQ): a new method for speech quality assessment of telephone networks and codecs. In IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Vol. 2, pp. 749–752. Cited by: 1st item.
- [23] H. Siuzdak, F. Grötschla, and L. A. Lanzendörfer (2024) SNAC: multi-scale neural audio codec. In Audio Imagination: NeurIPS 2024 Workshop AI-Driven Speech, Music, and Sound Generation, Note: arXiv:2410.14411 Cited by: §I.
- [24] C. H. Taal, R. C. Hendriks, R. Heusdens, and J. Jensen (2011) An algorithm for intelligibility prediction of time-frequency weighted noisy speech. IEEE Transactions on Audio, Speech, and Language Processing 19 (7), pp. 2125–2136. Cited by: 1st item.
- [25] A. van den Oord, O. Vinyals, and K. Kavukcuoglu (2017) Neural discrete representation learning. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 30, pp. 6306–6315. Cited by: § II-A.
- [26] C. Wang, S. Chen, Y. Wu, Z. Zhang, L. Zhou, S. Liu, Z. Chen, Y. Liu, H. Wang, J. Li, L. He, S. Zhao, and F. Wei (2023) Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXiv:2301.02111. Cited by: §I.
- [27] D. Xin, X. Tan, S. Takamichi, and H. Saruwatari (2024) BigCodec: pushing the limits of low-bitrate neural speech codec. arXiv preprint arXiv:2409.05377. Cited by: §I, § II-A.
- [28] D. Yang, S. Liu, R. Huang, J. Tian, C. Weng, and Y. Zou (2023) HiFi-Codec: group-residual vector quantization for high fidelity audio codec. arXiv preprint arXiv:2305.02765. Cited by: §I, § II-A.
- [29] Z. Ye, P. Sun, J. Lei, H. Lin, X. Tan, Z. Dai, Q. Kong, J. Chen, J. Pan, Q. Liu, Y. Guo, and W. Xue (2024) Codec does matter: exploring the semantic shortcoming of codec for audio language model. arXiv preprint arXiv:2408.17175. Cited by: §I.
- [30] N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi (2022) SoundStream: an end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing 30, pp. 495–507. External Links: Document Cited by: §I, § II-A, § III-A.
- [31] X. Zhang, D. Zhang, S. Li, Y. Zhou, and X. Qiu (2024) SpeechTokenizer: unified speech tokenizer for speech large language models. In International Conference on Learning Representations (ICLR), Cited by: §I, § II-B.