# A Closer Look at Neural Codec Resynthesis: Bridging the Gap between Codec and Waveform Generation
> Yuan Gong did this work at MIT, now he is with xAI Corp.
main.bib
## Abstract
Neural Audio Codecs, initially designed as a compression technique, have gained more attention recently for speech generation. Codec models represent each audio frame as a sequence of tokens, i.e., discrete embeddings. The discrete and low-frequency nature of neural codecs introduced a new way to generate speech with token-based models. As these tokens encode information at various levels of granularity, from coarse to fine, most existing works focus on how to better generate the coarse tokens. In this paper, we focus on an equally important but often overlooked question: How can we better resynthesize the waveform from coarse tokens? We point out that both the choice of learning target and resynthesis approach have a dramatic impact on the generated audio quality. Specifically, we study two different strategies based on token prediction and regression, and introduce a new method based on Schrödinger Bridge. We examine how different design choices affect machine and human perception. Audio demo page: https://alexander-h-liu.github.io/codec-resyn.github.io/
## 1 Introduction
Neural Codec models [zeghidour2021soundstream, defossez2022high, kumar2024high] initially emerged as compression techniques for audio compression. Despite being originally proposed for compression, neural audio codec models significantly impacted speech and audio modeling [wu2024towards] due to their discrete and low-frequency nature. Having a tokenized representation of audio introduces many benefits. For example, token-based modeling approaches similar to language models can be adopted for audio generation: VALL-E [wang2023neural], Speech-X [wang2023speechx], AudioLM [borsos2023audiolm], MusicLM [agostinelli2023musiclm] and MusicGEN [copet2024simple], just to name a few. Besides audio generation tasks, audio codecs can also be applied to cross-modality applications, e.g., making audio and large language model (LLM) integration seamless [chen2023lauragpt].
As shown in Fig. 1, codec models are trained to compress audio into discrete tokens at a low frequency rate to reduce the cost of transmission and storage. Formally, an encoder would first encode a slice (typically around 10 to 20ms) of signal $s$ into a latent $d$ -dimensional embedding $z∈ℝ^d$ . $z$ will then be iteratively quantized through $N$ Residual Vector Quantization [juang1982multiple, vasuki2006review] (RVQ) layers into $x_1,x_2,...,x_N$ where
$$
x_i=\operatorname*{\operatorname*{arg min}}_q∈ Q_{i}\lVert(\underbrace{
z-∑_j=1^i-1x_j)}_residual)-q\rVert, \tag{1}
$$
and $Q_i∈ℝ^V× d$ is the codebook containing $V$ codes ( $d$ -dimensional vectors) of the $i$ -th RVQ layer. Notice how each quantized embedding $x_i$ is a code within the corresponding codebook $Q_i$ . The input audio can therefore be compressed into a sequence of discrete variables which are the indices of the codes in their corresponding codebook. The bandwidth of neural audio codec models can be controlled by varying the number of codebooks $N$ and the size of each codebook $V$ . The goal of audio codec models is to restore the input signal with the quantized embedding and a decoder such that $s≈Decoder(∑_i=1^Nx_i).$
Due to the hierarchical structure of RVQ layers, the information carried by the first layer RVQ code $x_1$ is at a coarse level In practice, the set of coarse tokens can be defined as the first $c$ RVQ codes $x_1:c$ . This work considers the most common case $c=1$ without loss of generality since all methods can be extended to $c>1$ ., and that of the remaining layers $x_2:N$ gradually becomes more fine-grained [borsos2023audiolm]. Recent speech generation models [wang2023neural, wang2023speechx, borsos2023audiolm, chen2023lauragpt, wang2023viola, rubenstein2023audiopalm] have chosen to prioritize generating the coarse embedding $x_1$ . With the generated coarse embedding $x_1$ , the final step to synthesize audio is treated as a separate follow-up question and has received less attention. Solutions are often ad-hoc, for example, training a coarse-to-fine codec predictor with task-specific information like text and enrolled audio recordings [wang2023neural, wang2023speechx], or building a text-and-audio-conditioned codec vocoder [chen2023lauragpt].
In this work, we aim to study a question that has been overlooked in codec-based speech generation thus far – How to resynthesize speech using only the coarse representation? We focus on unconditional resynthesis that assumes only the coarse embedding $x_1$ is available, and no other task-specific information (such as transcription, speaker, or audio prompt) of the target speech is given. This assumption allows us to develop general methods not restricted to tasks or data annotation. We refer to the problem as Codec Resynthesis since the ultimate goal is to resynthesize audio from limited codes. Starting from coarse-to-fine resynthesis, we take a deep dive into unconditional codec resynthesis. With the insight into the learning target, we show how regressing continuous embedding instead of tokens is better. We further improve the modeling approach, introducing a discrete-to-continuous Codec Schrödinger Bridge. Finally, we present the strengths as well as the limitations of different methods, along with more challenges in codec resynthesis.
<details>
<summary>x1.png Details</summary>
### Visual Description
## Diagram: Neural Audio Compression and Codec Resynthesis Workflow
### Overview
The image depicts a two-stage process for neural audio compression and resynthesis. The left side illustrates **Neural Audio Compression**, while the right side shows **Codec Resynthesis**. Arrows indicate data flow, and components are labeled with technical terms like "RVQ" (Residual Vector Quantization) and "Schrödinger Bridge".
### Components/Axes
#### Left Diagram (Neural Audio Compression):
- **Input**: Audio waveform (top-left).
- **Encode**: Arrow pointing to latent variable **Z** (central node).
- **Decode**: Arrow from **Z** back to audio waveform (top-right).
- **Audio Codecs**: Three blocks labeled **x₁**, **x₂**, **x₃** (representing audio tokens).
- **RVQ**: Arrows connecting **x₁** → **x₂** → **x₃**, with labels "coarse tokens" (left) and "fine tokens" (right).
- **Dashed Arrow**: From **x₃** to **Z**, indicating iterative refinement.
#### Right Diagram (Codec Resynthesis):
- **Coarse-to-fine**: Red arrows forming a cycle between **x₁**, **x₂**, **x₃**, and **Z**.
- **Schrödinger Bridge**: Blue arrow connecting **x₃** to **Z** and back to **x₁**.
- **Output**: Reconstructed audio waveform (bottom).
### Detailed Analysis
- **Neural Audio Compression**:
- Audio is encoded into latent variable **Z** via RVQ, which quantizes audio tokens into coarse and fine representations.
- Decoding reconstructs the waveform from **Z**, with iterative refinement via the dashed arrow.
- **Codec Resynthesis**:
- **Coarse-to-fine** (red arrows): Iterative refinement of audio tokens (**x₁** → **x₂** → **x₃**) using **Z** as a latent anchor.
- **Schrödinger Bridge** (blue arrow): A probabilistic mapping between **x₃** and **Z**, ensuring fidelity in reconstruction.
### Key Observations
1. **Token Hierarchy**: Coarse tokens (**x₁**) are refined into fine tokens (**x₃**) via RVQ, balancing compression efficiency and quality.
2. **Latent Variable Role**: **Z** acts as a bridge between compression and resynthesis, enabling iterative optimization.
3. **Dual Arrows**: Red (coarse-to-fine) and blue (Schrödinger Bridge) arrows suggest complementary processes for high-fidelity reconstruction.
### Interpretation
This diagram illustrates a hybrid approach to neural audio compression:
- **Compression**: Audio is quantized into hierarchical tokens (coarse → fine) and encoded into a latent space (**Z**), enabling efficient storage.
- **Resynthesis**: The Schrödinger Bridge ensures probabilistic alignment between latent codes and tokens, while coarse-to-fine refinement restores audio quality.
- **Technical Implications**: The use of RVQ and iterative refinement suggests a focus on balancing compression ratios with perceptual quality, leveraging neural methods for both stages.
No numerical data or trends are present; the diagram emphasizes workflow and component relationships.
</details>
Figure 1: (Left) Illustration of neural codec compression with $N=3$ . RVQ stands for Residual Vector Quantization. (Right) Codec resynthesis methods that generates audio from the first RVQ code $x_1$ .
<details>
<summary>x2.png Details</summary>
### Visual Description
## Line Graph: Performance Metrics vs Decoder Input
### Overview
The image depicts a line graph comparing two performance metrics (SI-SNR and ESTOI) across sequential decoder inputs labeled from **x₁** to **z**. Two lines are plotted: a blue line representing **SI-SNR** and a green line representing **ESTOI**. Both metrics show an upward trend as decoder input complexity increases, with SI-SNR consistently outperforming ESTOI.
---
### Components/Axes
- **X-Axis (Decoder Input)**:
Labels include **x₁**, **x₁:2**, **x₁:3**, **x₁:4**, **x₁:5**, **x₁:6**, **x₁:7**, **x₁:8**, and **z**. These likely represent incremental decoder input configurations or stages.
- **Y-Axes**:
- **Left Y-Axis (SI-SNR)**: Ranges from **-5** to **5** in increments of 5.
- **Right Y-Axis (ESTOI)**: Ranges from **0.6** to **0.8** in increments of 0.1.
- **Legend**:
Located on the right side of the graph.
- **Blue line**: SI-SNR (Signal-to-Interference-plus-Noise Ratio).
- **Green line**: ESTOI (Extended Short-Time Objective Intelligibility).
- **Data Points**:
Marked with circular symbols (○) at each x-axis label.
---
### Detailed Analysis
1. **SI-SNR (Blue Line)**:
- Starts at **-5** for **x₁** and increases steadily to **5** at **z**.
- Notable jumps occur between **x₁:2** (≈0) and **x₁:3** (≈3).
- Slope is steeper in the latter half of the graph.
2. **ESTOI (Green Line)**:
- Begins at **0.6** for **x₁** and rises to **0.8** at **z**.
- Growth is more gradual compared to SI-SNR, with smaller increments (e.g., **x₁:3** ≈ 0.65, **x₁:7** ≈ 0.75).
- ESTOI remains below SI-SNR throughout, with both metrics reaching their respective maxima at **z**.
3. **Trends**:
- Both metrics improve monotonically with increasing decoder input.
- SI-SNR consistently outperforms ESTOI, as they are measured on different scales.
---
### Key Observations
- **Performance Correlation**: Higher decoder input complexity (e.g., **x₁:8** to **z**) correlates with significant improvements in both metrics.
- **Metric Divergence**: SI-SNR outperforms ESTOI, suggesting it may be a more sensitive indicator of signal quality.
- **Saturation Point**: Both metrics reach their maximum values at **z**, indicating potential saturation or diminishing gains at higher input levels.
---
### Interpretation
The graph demonstrates that decoder input complexity directly impacts signal quality metrics. SI-SNR’s steeper growth suggests it captures finer details of signal degradation, while ESTOI’s slower rise may reflect its focus on perceptual speech intelligibility. The saturation at **z** (both metrics at their maxima) implies that beyond a certain input complexity, further improvements yield marginal benefits for both metrics. This aligns with principles of signal processing, where optimal performance often plateaus due to hardware or algorithmic limitations.
**Note**: All values are approximate, derived from visual estimation of the plotted points.
</details>
Figure 2: SI-SNR and ESTOI [jensen2016algorithm] of decoded audio. See § A.1 for data and metric description.
## 2 Neural Codec Resynthesis
We consider codec resynthesis at the sequence level with non-autoregressive models, i.e., generating complete speech from the sequence of discrete embeddings $(x_1^(1),...,x_1^(l),...,x_1^(L))$ where $L$ is the sequence length. We shorthand $x_i≡ x_i^(1:L)$ hereafter for simplicity.
Coarse-to-fine Resynthesis. Due to the hierarchical structure of RVQ layers, each RVQ code $x_i$ depends on all the codes from prior layers $x_1:i-1$ . A simple method for codec resynthesis is therefore to iteratively predict the RVQ codes $x_2:N$ given the first $x_1$ . This can be achieved by training a model, parameterized by $θ$ , to maximize
$$
∑_i=2^Np_θ(x_i|x_1:i-1,i) \tag{2}
$$
where $p_θ(x_i|·)$ is a categorical distribution over codebook $Q_i$ . The process can be viewed as a coarse-to-fine prediction since the later RVQ codes encode fine-grained audio information. We note that similar coarse-to-fine models have been studied in prior works where they are autoregressive [borsos2023audiolm] or text-and-audio-conditioned [wang2023neural, wang2023speechx].
Is predicting the remaining RVQ codes $x_2:N$ necessary? While it is reasonable to resynthesize speech in a coarse-to-fine manner, modeling codes from all RVQ layers introduces a multi-task learning overhead during training and the risk of error propagation during inference. Prior work [copet2024simple] attempted to address the issue through parallel prediction but found coarse-to-fine prediction worked best empirically.
To find an alternative, we take a closer look at the quality of the audio decoded from representations at different layers of the codec model. Results are shown in Fig. 2. As expected, audio quality gradually improved when involving more fine-grain embeddings. However, a key observation is that the pre-quantized embedding $z$ , although never used as decoder input during training, yields the best audio quality. Since the ultimate goal is to generate high-fidelity audio, this observation suggests that predicting the remaining RVQ codes $x_2:N$ may not be necessary.
One-step Resynthesis. In light of our previous finding, we can train a one-step resynthesis model that simply predicts $z$ directly from $x_1$ through a regression model $f_θ$ that minimizes
$$
\lVert f_θ(x_1)-z\rVert. \tag{3}
$$
We refer to this regression-based method as one-step resynthesis since projecting $x_1$ to $z$ requires only a single forward pass of the model, and the result can be directly applied for decoding. Conversely, the coarse-to-fine method requires $N-1$ iterations to acquire $x_2:N$ . Prior work has found one-step resynthesis to be beneficial with audio and text conditions available [chen2023lauragpt], but it is unclear whether unconditional resynthesis is possible.
Codec Schrödinger Bridge Resynthesis. Although one-step generation sounds appealing, recent trends in generative models suggested iterative models, such as diffusion model [ho2020denoising], tend to synthesize data of better quality. From the output audio of one-step resynthesis (see demo page), we also find it results in robotic-sounding artifacts in the speech. This motivated us to explore iterative methods operating in the continuous embedding space to learn the mapping between $x_1$ and $z$ .
The Schrödinger Bridge (SB) problem [schrodinger1932theorie] aimed to find the entropy-regularized optimal transport between two arbitrary distributions, $p_x_0$ and $p_x_1$ . The solution to SB can be expressed by the following forward (4a) and backward (4b) stochastic differential equations (SDEs) [chen2021likelihood]:
$$
\displaystyle{\textnormal{d}}x_t \displaystyle=[f_t+β_t~{}{∇\log}{Ψ}_t]{dt}+{√{
β_t}}{\textnormal{d}}W_t,x_0∼ p_x_0, \displaystyle{\textnormal{d}}x_t \displaystyle=[f_t-β_t~{}{∇\log}\hat{{Ψ}}_t]{dt}+{
√{β_t}}{\textnormal{d}}\overline{W}_t,x_1∼ p_x_1,
$$
where the $T$ -step stochastic process is represented by $x_t$ with $t∈\{0,\frac{1}{T},\frac{2}{T},...,1\}$ , $f_t(x_t)$ is the linear drift, $β_t∈ℝ^d$ is the diffusion, $W_t,\overline{W}_t∈ℝ^d$ are the standard and reversed Wiener process. The terms ${∇\log}{Ψ}_t(x_t)$ and ${∇\log}\hat{{Ψ}}_t(x_t)$ are the forward and backward non-linear drifts for SB with $Ψ,\hat{{Ψ}}$ being the solution to the following coupled PDEs
$$
\begin{cases}\frac{∂Ψ}{∂ t}=-∇Ψ^⊤f-\frac{1}{2}
βΔΨ\\
\frac{∂\hat{{Ψ}}}{∂ t}=-∇·(\hat{{Ψ}}f)+\frac{1}{2
}βΔ\hat{{Ψ}}\end{cases}s.t.Ψ_0\hat{{Ψ}}_0=p_0,Ψ_1
\hat{{Ψ}}=p_1. \tag{5}
$$
SB provides a general mathematical framework for distribution mapping, but solving it can be challenging in practice since $Ψ$ and $\hat{{Ψ}}$ are often intractable in real world applications.
Fortunately, prior work has shown that SB can be tractable for certain applications where paired data of the two distributions is available [liu2023i2sb]. By setting $f:=0$ (merging linear drift into non-linear drift) and $\hat{{Ψ}}_0:=δ_x_0$ (Dirac delta distribution centered around $x_0$ ), a neural network $ε_θ$ for estimating the score function ${∇\log}\hat{{Ψ}}_t$ of the backward SDE (4b) can be trained through minimizing
$$
\lVertε_θ(x_t,t)-\frac{x_t-x_0}{σ_t}\rVert,~{}~{}x_t∼N(~{}\frac{\bar{σ}_t^2}{\bar{σ}_t^2+{σ
}_t^2}x_0+\frac{{σ}_t^2}{\bar{σ}_t^2+{σ}_t^2}
x_1~{},~{}\frac{{σ}_t^2\bar{σ}_t^2}{\bar{σ}_t^2+{
σ}_t^2}· I~{}), \tag{6}
$$
with $σ^2_t{:=}∫_0^tβ_τ{\textnormal{d}}τ$ and $\bar{σ}^2_t{:=}∫_t^1β_τ{\textnormal{d}}τ$ .
Table 1: Codec resynthesis results. All metrics are the higher the better except WER, see § A.1 for detailed explanation. Best results are bolded. Number of function evaluations (NFE) reflects the number of forward passes required for synthesis.
| Decoder input | NFE | Intrusive Metrics | | Perceptual Metrics | | | | |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| SI-SNR ( $↑$ ) | ESTOI ( $↑$ ) | ViSQOL ( $↑$ ) | | WER(%; $↓$ ) | SIM ( $↑$ ) | MOS ( $↑$ ) | | |
| Baseline | | | | | | | | |
| 1 ${}^st$ RVQ code $x_1$ | | -5.39 | 0.56 | 2.99 | | 24.7 | 0.217 | - |
| Resynthesis Methods | | | | | | | | |
| Coarse-to-fine | 7 | -3.09 | 0.71 | 3.36 | | 22.2 | 0.469 | 3.19 $±$ 0.12 |
| One-step regression | 1 | -1.12 | 0.75 | 3.52 | | 11.5 | 0.495 | 3.06 $±$ 0.13 |
| Schrödinger Bridge | 1 | -1.38 | 0.74 | 3.49 | | 14.5 | 0.491 | 2.95 $±$ 0.14 |
| 4 | -1.55 | 0.74 | 3.47 | | 18.3 | 0.507 | - | |
| 7 | -1.90 | 0.73 | 3.42 | | 19.7 | 0.506 | 3.43 $±$ 0.11 | |
| 16 | -2.30 | 0.72 | 3.37 | | 21.1 | 0.501 | 3.46 $±$ 0.11 | |
| 32 | -2.52 | 0.71 | 3.33 | | 22.2 | 0.493 | - | |
| Topline | | | | | | | | |
| 8 RVQ code $x_1:8$ | | 4.34 | 0.88 | 4.27 | | 2.7 | 0.861 | - |
| Pre-quantize emb. $z$ | | 4.87 | 0.95 | 4.55 | | 2.7 | 0.922 | - |
| Ground Truth | | - | 1.00 | 5.00 | | 2.4 | 1.000 | 3.74 $±$ 0.11 |
For codec resynthesis, we are interested in transporting between the distribution $p_x_0≡ p_z$ of pre-quantized embedding $x_0≡ z$ and the first RVQ code distribution $p_x_1$ . Since the pair relation $(x_0,x_1)$ is available through the audio encoding process, Codec Schrödinger Bridge can be trained directly with Eq. 6. During inference, Codec Schrödinger Bridge can be used to construct the backward SDE (Eq. 4b) and derive pre-quantized embedding $x_0$ from $x_1$ using DDPM [ho2020denoising]. The backward process can be simulated with different step sizes by breaking down the schedule from $t=1$ to $t=0$ into more/less segments, traversing iteratively from $x_1$ to $x_0$ . In practice, smaller step sizes result in better generation quality [de2021diffusion, chen2021likelihood, liu2023i2sb] at a cost of more forward passes through the model.
## 3 Experiments
Due to space constraints, details on model architecture, training hyperparameters, datasets, and evaluation metrics are provided in Appendix Section A.1. In Table 1, we report resynthesis results on LibriSpeech [panayotov2015librispeech] test-clean. The performance baseline is obtained from audio decoded from the first RVQ code $x_1$ without resynthesis. For resynthesis methods, the decoder of Encodec takes the resynthesis model output as input. Different toplines should be considered for different methods: (1) decoding with full RVQ codes $x_1:8$ , which is the topline for coarse-to-fine method; (2) decoding with pre-quantized embedding $z$ , which is the topline for one-step regression and Schrödinger Bridge. Ground truth is the raw audio used as the reference for intrusive metrics.
The pre-quantized embedding is consistently a better target. As in the findings shown in Fig 2, the results in Table 1 indicate that pre-quantized embedding $z$ not only provides a higher performance upper bound (see toplines), but also results in better models when used as a learning target. The one-step regression model performs better on all intrusive metrics at a significantly lower inference cost (NFE=1). In addition, Schrödinger Bridge consistently performs better than the coarse-to-fine model in both objective and subjective metrics at the same (NFE=7) or lower (NFE=4) inference cost. In short, pre-quantized representations of codec models are better than tokens for resynthesis.
A good objective score does not imply better audio quality. It is worth noting that the one-step regression model is considered the best model in terms of WER and all intrusive metrics in Table 1, including SI-SNR and ViSQOL that are commonly adopted for codec model development [defossez2022high, kumar2024high]. However, this result contradicts human perception as reflected by MOS. In fact, resynthesized speech from one-step regression exhibited a lot of artificial sounding and robotic voices (see audio samples on the demo page), resulting a similar MOS to the coarse-to-fine model. In contrast, Codec Schrödinger Bridge significantly reduced the artifacts when taking a smaller step size (more NFE as a trade-off), resulting in more natural output as reflected by MOS. This finding suggests that the well-known concept that denoising methods like diffusion [ho2020denoising] are strong in generating high-fidelity data also holds for codec resynthesis.
<details>
<summary>x3.png Details</summary>
### Visual Description
## Line Graph: Performance Metrics vs. Number of Function Evaluations (NFE)
### Overview
The image is a dual-axis line graph comparing three optimization strategies (Coarse-to-fine, Schrödinger Bridge, Input) across two metrics: Word Error Rate (WER) and Structural Similarity Index (SIM). The x-axis represents the number of Function Evaluations (NFE) from 0 to 7. The graph shows how each strategy's performance evolves with increasing computational effort.
### Components/Axes
- **X-axis**: Number of Function Evaluations (NFE), integer values from 0 to 7.
- **Left Y-axis (WER)**: Word Error Rate, linear scale from 0.15 to 0.25.
- **Right Y-axis (SIM)**: Structural Similarity Index, linear scale from 0.3 to 0.5.
- **Legend**: Located in the bottom-right corner, with three entries:
- **Blue line with circles**: Coarse-to-fine
- **Green line with circles**: Schrödinger Bridge
- **Gray line with diamond**: Input
### Detailed Analysis
#### Coarse-to-fine (Blue)
- **NFE 0**: WER = 0.25, SIM = 0.3
- **NFE 1**: WER ≈ 0.18, SIM ≈ 0.4
- **NFE 2**: WER ≈ 0.19, SIM ≈ 0.42
- **NFE 3**: WER ≈ 0.20, SIM ≈ 0.44
- **NFE 4**: WER ≈ 0.21, SIM ≈ 0.45
- **NFE 5**: WER ≈ 0.22, SIM ≈ 0.46
- **NFE 6**: WER ≈ 0.22, SIM ≈ 0.47
- **NFE 7**: WER ≈ 0.23, SIM ≈ 0.48
#### Schrödinger Bridge (Green)
- **NFE 0**: WER = 0.25, SIM = 0.5
- **NFE 1**: WER ≈ 0.15, SIM ≈ 0.5
- **NFE 2**: WER ≈ 0.17, SIM ≈ 0.5
- **NFE 3**: WER ≈ 0.18, SIM ≈ 0.5
- **NFE 4**: WER ≈ 0.19, SIM ≈ 0.5
- **NFE 5**: WER ≈ 0.19, SIM ≈ 0.5
- **NFE 6**: WER ≈ 0.20, SIM ≈ 0.5
- **NFE 7**: WER ≈ 0.20, SIM ≈ 0.5
#### Input (Gray)
- **NFE 0**: WER = 0.25, SIM = 0.3
- **NFE 1**: WER ≈ 0.15, SIM ≈ 0.5
- **NFE 2–7**: No data points (line remains flat at WER = 0.15, SIM = 0.5)
### Key Observations
1. **Input Strategy**:
- Drops sharply from (WER=0.25, SIM=0.3) at NFE=0 to (WER=0.15, SIM=0.5) at NFE=1.
- Remains constant for NFE ≥ 1, suggesting no further improvement with additional evaluations.
2. **Coarse-to-fine**:
- Initial WER decrease (0.25 → 0.18) followed by gradual WER increase (0.18 → 0.23) as NFE increases.
- SIM improves steadily (0.3 → 0.48) with increasing NFE.
3. **Schrödinger Bridge**:
- WER decreases sharply (0.25 → 0.15) at NFE=1, then plateaus.
- SIM remains constant at 0.5 across all NFE values.
### Interpretation
- **Trade-off Dynamics**:
- Coarse-to-fine shows a trade-off between WER and SIM, with WER increasing as SIM improves at higher NFE.
- Schrödinger Bridge achieves optimal SIM (0.5) immediately but fails to reduce WER further after NFE=1.
- Input strategy demonstrates a one-time "jump" in performance, after which it stagnates, implying limited scalability.
- **Efficiency Insights**:
- Schrödinger Bridge achieves the lowest WER (0.15) with minimal NFE (1), but its WER plateaus.
- Coarse-to-fine requires more NFE to improve SIM but suffers from WER degradation at higher NFE.
- Input's flat performance after NFE=1 suggests it may represent a baseline or non-iterative method.
- **Anomalies**:
- The Input strategy's abrupt drop at NFE=1 contrasts with the gradual changes in other methods.
- Schrödinger Bridge's constant SIM across all NFE values is unusual, indicating potential saturation or a fixed upper bound.
This graph highlights the differing optimization trajectories of the three methods, with Coarse-to-fine showing the most dynamic behavior and Input/Schrödinger Bridge exhibiting early saturation.
</details>
Figure 3: Word error rate (WER) and speaker similarity (SIM) w.r.t. different NFEs for iterative methods.
Table 2: Codec resynthesis result with different amounts of training data. LibriLight [librilight] (LL) contains more than 60k hours of speech. It is more than 62.5x larger than LibriSpeech [panayotov2015librispeech] (LS) which contains 960 hours of speech only.
| Decoder input | Training data | | | | |
| --- | --- | --- | --- | --- | --- |
| LL (60k hours) | | LS (960 hours) | | | |
| WER(%) | SIM | | WER(%) | SIM | |
| Coarse-to-fine | 22.2 | 0.469 | | 24.6 | 0.435 |
| One-step | 11.5 | 0.495 | | 28.8 | 0.233 |
| SB (NFE=1) | 14.5 | 0.491 | | 16.0 | 0.482 |
| SB (NFE=7) | 19.7 | 0.506 | | 22.7 | 0.485 |
Iterative methods improve sound quality, not content. Next, we are interested in finding out the reason why iterative methods provided a better MOS yet worse WER. Interestingly, we found that speech intelligibility (as measured by Whisper) does not increase as a function of NFE as shown in Fig. 3. This is surprising since single NFE degenerates these iterative methods into one-step methods. For the coarse-to-fine model, single NFE predicts only the second RVQ code $x_2$ , which suggests that the (predicted) fine-grain representation $x_3:8$ actually introduces more noise than signal with regard to content. Single NFE Schrödinger Bridge is conceptually the same as the one-step regression model, which results in 14.5% WER that is closer to that of the latter 11.5% (Table 1). This indicates that the content of speech is harder to preserve through a more sophisticated backward process. We note that the problem could potentially be solved with ad-hoc model design for specific applications, e.g., conditioning the coarse-to-fine model with phone sequence [wang2023neural]. On the other hand, speaker similarity generally improved as NFE increased, and we hypothesize that naturalness plays an important role. The difference in reference-free MOS between different NFEs with Schrödinger Bridge in Table 1 also supported this hypothesis.
Iterative methods are less prone to overfitting. In Table 2, we report the results of training models on LibriSpeech (960 hour) instead of LibriLight (60k hour) to observe model behavior when massive training data is not available. Surprisingly, the one-step method suffered significantly from the lack of training data, resulting in the worst WER and SIM. In contrast, the coarse-to-fine and Schrödinger Bridge models have significantly less loss in performance. In practice, we also found that the one-step regression model overfit the smaller training set after 300k updates (best result reported), whereas both the coarse-to-fine and Schrödinger Bridge model consistently improved throughout the 500k training steps. These results suggest that iterative methods that partition the task into smaller subtasks are less prone to overfitting and more robust to low-resource scenarios.
## 4 Conclusion
Summarizing different methods. We first conclude that among the methods explored in this work, coarse-to-fine generation is the least ideal method with poor audio quality and high inference cost. The one-step regression model is efficient and effective in subjective metrics but results in worse objective scores and robustness. Codec Schrödinger Bridge falls slightly behind but can be particularly useful for generating high-fidelity audio. Finally, we note that while a clear improvement is made by all three methods over the baseline, results suggest that these methods still have much room for improvement.
Rethinking the evaluation metrics. We note that the ultimate goal of codec resynthesis is finding the mapping between a coarse-level codec and realistic audio, which can be a one-to-many mapping. In other words, any resynthesized codec embedding that decodes into realistic audio should be considered a success, the “ground truth” should only be used for training but not evaluation. This is especially true for applications like text-to-speech. Therefore, intrusive metrics are not ideal for the task. While reference-free machine perceptual metrics like WER can be a good alternative, it does not reflect human preference. Finding a better automatic subjective metric that better reflects human preference is still an important open question.
Revisiting token-based and bridge-style generation. Recent token-based models [wang2023neural, wang2023speechx] and bridge-style models [le2024voicebox, liu2023generative, vyas2023audiobox] each have their strengths in speech generation. While they appear as two distinct concepts, our findings suggest it is possible to combine the advantages of the two different methods, as demonstrated by the proposed Codec Schrödinger Bridge.
Limitations. We left exploring the generalizability of codec resynthesis as an important future work. For example, the multilingual setup is expected to be even more challenging. Besides speech, sound and music are also common uses of audio codec models. Generalizing codec resynthesis beyond speech can potentially reduce the burden of more audio generative models, e.g., music generation model [copet2024simple] that have to autoregressively generate codec both left-to-right and course-to-fine.
Acknowledgments. The authors would like to thank Guan-Horng Liu for the helpful discussion, Roham Mehrabi and Pei-Ling Chiang for setting up human evaluation on AmazonTurk.
## Appendix A Appendix
### A.1 Experiment Setup
Model. The 6kbps Encodec [defossez2022high] is used for the codec model with $d=128$ dimensional embedding at 75Hz, $N=8$ RVQ layers each with a codebook of size $V=1024$ . For all three methods, the input is the discrete embedding $x_1$ of the first Encodec RVQ layer. We trained a 12-layer Transformer Encoder [vaswani2017attention] with a 16-head self-attention, an embedding dimension of 1024/4096 for self-attention/feedforward layers, and a 0.05 layer dropout. For the coarse-to-fine model, we used a learnable stage embedding to encode the state $i$ . For the Schrödinger Bridge model, we used sinusoidal positional embedding [vaswani2017attention] to encode the timestep $t$ . Adaptive LayerNorm [xu2019understanding] is used to normalize the output of each layer conditioning on the stage or time embedding. We used the Adam optimizer with a weight decay of 0.01 to train each model for 1M steps, with learning ramping up in 32k steps and then linearly decaying to 0 for the remaining steps. The peak learning rate for each method is swept, 1e-4/5e-4/5e-4 is used for the coarse-to-fine/one-step/SB model based on validation loss. Models have around 210M parameters, and training takes 9 days on 4 A6000 GPUs. For the Schrödinger Bridge model, we set $T=1000$ and $β_t$ followed a symmetric noise scheduling [de2021diffusion] peaking at 0.3. Additionally, we found that conditioning the model on initial point $x_1$ (by projecting and adding it to the Transformer input) regardless of $t$ helpful.
Data. All models are trained on LibriLight [librilight], an audiobook corpus with 60k hours of English speech. The dev-clean / test-clean subset of LibriSpeech [panayotov2015librispeech] is used for validation/evaluation respectively. Each audio is randomly cropped to equal length to form an 800-second batch.
Evaluation Metrics. To assess the quality of the resynthesized speech, the following metrics were used: Scale-Invariant Signal-to-Noise Ratio (SI-SNR); Extended Short-Time Objective Intelligibility (ESTOI; [jensen2016algorithm]); ViSQOL [chinen2020visqol], an intrusive perceptual metric that estimates mean opinion score based on spectral similarity; Word Error Rate (WER) comparing the recognition result of Whisper v2 [radford2023robust] on resynthesized speech versus ground truth transcriptions; Speaker Similarity (SIM), the cosine similarity between the speaker embedding of ground truth and resynthesized speech extracted by ECAPA-TDNN [desplanques2020ecapa, ravanelli2021speechbrain]; Subjective Mean Opinion Score (MOS) assessing audio naturalness and quality on a scale of 1 to 5, with increments of 1. We randomly selected 40 sentences from the test set and collected 10 ratings for each model for each sample. We followed the standard approach to collect ratings through AmazonTurk [naderi2020open, ribeiro2011crowdmos] with the reward of $0.05 per rating. Each worker rated 8 sentences and for each sentence audio from 6 different systems were presented. We reported the average rating for each system on a 95% confidence interval.