2505.24437

# SwitchCodec: A High-Fidelity Nerual Audio Codec With Sparse Quantization **Authors**: Jin Wang, Wenbin Jiang, Xiangbo Wang, Yubo You, Sheng Fang ## Abstract Neural audio compression has emerged as a promising technology for efficiently representing speech, music, and general audio. However, existing methods suffer from significant performance degradation at limited bitrates, where the available embedding space is sharply constrained. To address this, we propose a universal high-fidelity neural audio compression algorithm featuring Residual Experts Vector Quantization (REVQ), which substantially expands the embedding space with minimal impact on bandwidth. A gentle load-balancing strategy is introduced to ensure the full utilization of this expanded space. Furthermore, we develop a novel multi-tiered discriminator that periodically stratifies STFT spectra, guiding the generator to attend to both fine-grained details and the overall spectral structure. To support multiple bitrates without quality loss at the lower end, we adopt an efficient post-training strategy. The proposed codec achieves impressive performance, with PESQ and ViSQOL scores of 2.87 and 4.04, respectively, at 2.67 kbps bandwidth. On the perceptual side, subjective listening tests confirm that our model delivers higher audio quality compared to the baseline. Additionally, the approach effectively reduces spectral blur, decreasing the distance to the original mel-spectrogram by 13%. Finally, our post-training strategy achieves performance comparable to dedicated fixed-bitrate models while reducing the required training time by half. keywords: nerual audio compression, embedding space, discriminator, quantization journal: Knowledge-Based Systems [School_com]organization=School of Communication Engineering, addressline=Hangzhou Dianzi University, city=Hangzhou, postcode=310018, country=China [School_ele]organization=School of Electronics & Information , addressline=Hangzhou Dianzi University, city=Hangzhou, postcode=310018, country=China [Lab]organization=Intelligent Information Processing Lab, addressline=Hangzhou Dianzi University, city=Hangzhou, postcode=310018, country=China ## 1 Introduction <details> <summary>x1.png Details</summary>  ### Visual Description ## Diagram: Hybrid Autoencoder with Quantization and Discriminator ### Overview The diagram illustrates a hybrid neural network architecture combining an autoencoder with quantization, a router mechanism, and a discriminator. The system processes input data through an encoder, routes latent representations through quantized vectors, reconstructs output via a decoder, and evaluates authenticity using a discriminator. ### Components/Axes 1. **Encoder (E)** - Input: Raw data (orange waveform) - Output: Latent vectors `Z_e1`, `Z_e2`, `Z_e3` (colored blocks) 2. **Router** - Takes encoder outputs and distributes them to quantized vectors `Q1`, `Q2`, ..., `Qn` (blue blocks) - Includes a "Shared Quant." block (orange gradient) 3. **Add** - Combines quantized vectors (`Z_q1`, `Z_q2`, `Z_q3`) 4. **Decoder (D)** - Reconstructs output from combined quantized vectors - Output: Processed waveform (orange waveform) 5. **Discriminator** - Input: Reconstructed waveform - Output: Real/Fake classification via: - `2xT` Conv2d Layers - `n x Δf` Fully Connected Layers - `p x Δf'` Fully Connected Layers ### Detailed Analysis - **Encoder**: Processes input into latent representations (`Z_e1-Z_e3`). - **Router**: Splits latent vectors into quantized components (`Q1-Qn`), suggesting vector quantization for compression. - **Add**: Merges quantized vectors (`Z_q1-Z_q3`) for reconstruction. - **Decoder**: Reconstructs output waveform from quantized representations. - **Discriminator**: Uses convolutional (`Conv2d`) and fully connected layers to classify output as real or fake, indicating a GAN-like component for authenticity verification. ### Key Observations - **Quantization Flow**: Latent vectors are quantized (`Q1-Qn`) and shared, implying a compression mechanism. - **Hybrid Architecture**: Combines autoencoder (E-D) with GAN discriminator for quality control. - **Router Complexity**: Multiple quantized vectors (`Q1-Qn`) suggest dynamic routing or hierarchical quantization. ### Interpretation This architecture likely implements a **Vector Quantized Variational Autoencoder (VQ-VAE)** with a discriminator for enhanced reconstruction quality. The router’s quantization and shared vectors enable efficient compression, while the discriminator ensures outputs meet realism criteria. The use of `Conv2d` layers in the discriminator suggests spatial feature extraction, critical for waveform authenticity checks. The system balances compression (via quantization) and quality (via GAN supervision), typical in applications like audio synthesis or image generation. </details> Figure 1: The overall architecture of the proposed SwitchCodec. An input audio waveform is first segmented into windows. The encoder then maps each window to a structured latent representation, $Z_e$ . Quantization is performed through a dual-path process. As a foundational step, a shared quantizer provides an initial, coarse quantization for the entire latent vector $Z_e$ . In parallel, our key innovation, the REVQ module, uses a Router to dynamically assign a subset of routing quantizers ( $Q_1$ to $Q_n$ ) to capture adaptive features from $Z_e$ . The outputs from both the shared and routed quantization paths are then summed to form the final, refined latent representation $Z_q$ . The entire framework is optimized with a combination of reconstruction and adversarial losses (discriminator architecture detailed in Section 3.4). Audio codecs play a cruicial role in modern data transmission and storage by enabling efficient compression of audio signals while preserving perceptual quality. Traditional audio codecs have long relied on expertise in psycho-acoustics and signal processing to design handcrafted compression strategies, which exploit human auditory characteristics to discard imperceptible information. In contrast, neural audio codecs, leveraging data-driven approaches, learn efficient audio discrete representations via deep learning, achieving significant advancements in audio compression. In neural audio codecs based on the VQ-VAE [1] framework, input audio is first encoded into low-dimensional latent representations, which are quantized into discrete codes; these codes are then dequantized using one or multiple codebooks to reconstruct the latents for audio reconstruction. Due to the representation space being severely constrained by individual codebooks [2], which leads to poor model performance, SoundStream [3] introduces Residual Vector Quantization (Residual VQ) [4] to quantize the latent representation in a multi-stage manner. To enhance codebook utilization, HiFi-Codec [5] proposes Group RVQ, a quantization method fusing GVQ and RVQ, which performs multi-stage grouped quantization on latent and concatenates them. Y. Chae et al. [6] dynamically adjust the number of quantizers used for each frame of latent according to quantization difficulty, reducing bandwidth usage with minimal impact on reconstruction quality. However, when codebook space is insufficient, all such methods exhibit inaccuracy directly in the latent retrieval process, impairing the decoder’s ability to reconstruct high-fidelity audio. Thus, in the scenario of higher compression, these advanced models still exhibit audio artifacts such as tonal artifacts [7], pitch distortions, and periodicity anomalies [8] Similarly, under efficiency constraints, large language models (LLMs) suffer from underfitting and limited expressivity due to insufficient parameters. Shazeer [9] first introduced the Mixture of Experts (MoE) model into large language models, decoupling computational cost from model scale via sparse gating layers. Gshard [10] incorporated an auxiliary loss function to penalize expert load variance, forcing the router to distribute tokens uniformly. Switch Transformers [11] simplified routing by activating only one expert per token and presetting token processing limits for each expert. DeepSeek-v3 [12] introduced a load-balancing strategy without auxiliary loss, avoiding performance degradation caused by model parameter optimization being affected by auxiliary losses. These methods address both insufficient expert utilization and training instability, thereby establishing sparsity-based architectures as the dominant solution for large language models. Inspired by these approaches, we propose a novel sparse quantization method that adopts a shared quantizer to extract cross-audio universal information, while assigning tailored routed quantizers to each audio via a gate network to capture their audio-specific characteristics, thus achieving the decoupling of compression bitrate and codebook capacity. Additionally, we propose a discriminator that can better distinguish the spectrogram differences between the original audio segment and the generated one by periodically segmenting the STFT spectrum. Consequently, our contributions can be summarized as follows: - We propose Residual Experts Vector Quantization, a method that breaks through the codebook capacity limitations of previous quantization methods and demonstrates better performance under higher compression. - We propose Developing Router Protection Strategy (DRPS) that enhances the utilization of routed quantizers without excessively enforcing uniformity, thus avoiding performance degradation caused by over-averaging requirements. - We introduce a Multi-Tiered STFT Discriminator that segments spectrograms into hierarchical frequency bands, ensuring amplification of differences between generated and original spectrograms while avoiding excessive focus on spectral regions with low information density. - We propose a post-training strategy that minimizes performance discrepancies across the full bitrate range induced by quantizer dropout (used for multi-bitrate support), with minimal computational overhead. ## 2 Related Works ### 2.1 Nerual audio codecs End-to-end neural audio codecs abandon manually designed signal processing components in traditional audio codecs [13] - [15], and instead rely solely on data-driven methods to learn the intrinsic features of audio. Garbacea et al. [16] achieves audio compression at 1.6 kbps by conditioning a WaveNet-based model [17] on discrete units derived from a VQ-VAE [1] model. Jiang et al. replace the Straight-Through Estimator (STE) [39] with the Gumbel-Softmax [18] to propagate gradients from the encoder to the decoder [19, 20]. SoundStream [3] introduces a fully convolutional encoder-decoder architecture [21] integrated with Residual Vector Quantization (RVQ), enabling high-quality compression across multiple bitrates. Building upon SoundStream, EnCodec [22] enhances performance through proposed multiscale STFT discriminator and loss balancer. Following the training recipe of BigVGAN [23], DAC [24] integrates snake activation function [25] and quantization techniques [26] into a VQGAN-based model to enhance reconstruction quality. SD-Codec [27] employs joint learning of audio resynthesis and separation to explicitly map audio signals from distinct domains to separate codebooks. Applying prior state-of-the-art codecs to LLMs in TTS requires designing multi-sequence prediction [28], which introduces bottlenecks compromising both efficiency and robustness. To address this issue, Single-Codec [29] and Wavtokenizer [30] utilize only a single codebook while introducing more powerful encoders and decoders, thereby achieving better performance. However, single-codebook architectures can only support different bitrates by training the model multiple times, which consumes substantial computational resources. Although the RVQGAN model uses dropout to enable multi-bitrate support, it significantly degrades performance at low bitrates. To address these issues, we propose a method tailored to our framework, which substantially reduces the damage caused by dropout to reconstruction performance across full bandwidth at lower computational cost. ### 2.2 Vector Quantization Quantization is a lossy source coding technique, inherently incurring information loss [31]. Specifically, Vector Quantization (VQ) [32], as a powerful data compression method, maps high-dimensional input vectors to a discrete set of pre-defined representative vectors (codebook) to achieve efficient encoding. The quantization loss is primarily determined by the usage and capacity of the codebook. To mitigate this loss, VQ-VAE-2 [33] incorporate exponential moving average (EMA) updates, continuously adapting the codebook during training to better align it with the distribution of encoded latents. Dhariwal et al. [34] replace codebook vectors that remain unused for several batches with randomly sampled input frames from the current batch. SoundStream [3] initialize the codebook with centroids learned by running the k-means algorithm [35] on the first training batch, ensuring it is close to the input distribution. DAC [24] improves codebook utilization by compressing code embeddings into a low-dimensional space. Yao et al. [37] design a two-stage training process to enhance the utilization of larger codebook by selecting the most frequently used codebook vectors to form a optimized codebook. While prior methods aim to maximize codebook utilization, the limited codebook space becomes insufficient under high compression scenario. Consequently, it fails to adequately represent the encoded latents, resulting in substantial quantization error. To address this issue, we propose Residual Experts Vector Quantization (REVQ), which minimizes quantization error by sparsely activating appropriate quantizers to dynamically match the latent structure. ### 2.3 STFT-based discriminator The STFT-based Discriminator is designed to distinguish the spectra of original and decoded audio, guiding the generator to produce high-fidelity audio. Welker et al. [36] found that neural audio codecs produce unnatural audio and poor SI-SDR scores without adversarial losses. Kumar et al. [24] show through ablation studies that STFT-based discriminators significantly improve audio quality, whereas wave-based discriminators only slightly enhance model performance. UnivNet proposed the multi-resolution spectrogram discriminator (MRSD) [38] to improve the problems of spectrogram blurring and over - smoothing artifacts. Since MRSD only utilized the magnitude spectrogram, EnCodec [22] proposed a multi - scale complex STFT - based (MS - STFT) discriminator to enhance phase modeling. Building upon the MS-STFT, MRD [24] splits the STFT into sub-bands to amplify differences across frequency bands, slightly improving high-frequency prediction and mitigating aliasing artifacts. While differences in the high-frequency region are amplified in MRD, equally important sub-bands contain unequal information, leading the discriminator to over-focus on simple regions and neglect high-information areas. To address this, we propose multi-tiered STFT Discriminator, which splits frequency bands to enhance inter-band differences while ensuring comparable information density across each band. ## 3 Proposed Methods ### 3.1 Encoder-Decoder Architecture The model’s encoder adopts the hierarchical convolutional architecture of DAC [24], consisting of an input convolutional layer, four encoder blocks, and an output convolutional layer. The input signal is first processed by a 7 $×$ 1 convolutional kernel that expands to 64 channels. Four downsampling blocks follow, each containing three residual units and a downsampling convolution. The residual units leverage Snake1d activation functions and dilated convolutions for multi-scale feature extraction. The channel dimension progressively doubles from 64 to 1024, while the temporal dimension is compressed with ratios progressing from 1 $×$ to 512 $×$ . A final 3 $×$ 1 convolution outputs a 1024-dimensional feature vector. The decoder employs a symmetric architecture to the encoder, leveraging transposed convolutions for signal reconstruction. The 1024-dimensional input features are first expanded to 1536 channels via a 7 $×$ 1 convolution. Four upsampling blocks follow, each comprising transposed convolution upsampling, Snake1d activation, and three residual units. The channel dimension progressively halves from 1536 to 96, while the temporal dimension expands with ratios progressing from 1 $×$ to 512 $×$ . A final 7 $×$ 1 convolution with Tanh activation outputs the reconstructed signal, ensuring precise symmetric reconstruction with the encoder. Algorithm 1 Residual Experts Vector Quantization 0: $z=enc(x)$ the output of the encoder, shared quantizers $Q_i^(s)$ for $i=1..N_s$ , routing quantizers $Q_j^(r)$ , $mask_j$ for $j=1..N_r$ , 0: the quantized $\hat{z}$ 1: $\hat{z}← 0.0$ $residual←\hat{z}$ 2: for $i=1$ to $Ns$ do 3: $\hat{z}$ $+=$ $Q_i^(s)(residual)$ 4: residual $-=$ $Q_i^(s)(residual)$ 5: end for 6: for $j=1$ to $Nr$ do 7: $\hat{z}$ $+=$ $Q_j^(r)(residual)× mask_j$ 8: residual $-=$ $Q_j^(r)(residual)× mask_j$ 9: end for 10: return Outputs <details> <summary>Figure_2.png Details</summary>  ### Visual Description ## Heatmap: Quantile Value Analysis Across Quantization Levels ### Overview The image contains two vertically stacked heatmaps comparing quantile values (Q1 and Q3) across six quantization levels ("Original Z" to "Quant. 5"). Values are color-coded (red for negative, green/orange for positive) and labeled with precise numerical values. Both heatmaps share identical column labels but differ in value distributions. --- ### Components/Axes - **X-axis (Columns)**: - Labels: "Original Z", "Quant. 1", "Quant. 2", "Quant. 3", "Quant. 4", "Quant. 5" - Position: Bottom of both heatmaps - **Y-axis (Rows)**: - Labels: "Q1" (lower row), "Q3" (upper row) - Position: Left side of both heatmaps - **Color Coding**: - Red: Negative values (Q1) - Green/Orange: Positive values (Q3) - No explicit legend; colors inferred from value polarity --- ### Detailed Analysis #### First Heatmap (Top) | Column | Q3 Value | Q1 Value | Color | |--------------|----------|----------|-------------| | Original Z | 0.47 | -0.47 | Red/Green | | Quant. 1 | 0.37 | -0.37 | Blue | | Quant. 2 | 0.54 | -0.53 | Green | | Quant. 3 | 0.63 | -0.63 | Orange | | Quant. 4 | 0.63 | -0.63 | Dark Blue | | Quant. 5 | 0.63 | -0.63 | Dark Blue | #### Second Heatmap (Bottom) | Column | Q3 Value | Q1 Value | Color | |--------------|----------|----------|-------------| | Original Z | 0.47 | -0.47 | Red/Green | | Quant. 1 | 0.37 | -0.37 | Blue | | Quant. 2 | 0.37 | -0.37 | Dark Blue | | Quant. 3 | 0.37 | -0.37 | Dark Blue | | Quant. 4 | 0.49 | -0.48 | Green | | Quant. 5 | 0.51 | -0.52 | Orange | --- ### Key Observations 1. **Trend Verification**: - **First Heatmap**: - Q3 values increase monotonically from 0.47 (Original Z) to 0.63 (Quant. 3-5). - Q1 values decrease monotonically from -0.47 (Original Z) to -0.63 (Quant. 3-5). - **Second Heatmap**: - Q3 values plateau at 0.37 (Quant. 1-3) before rising to 0.49 (Quant. 4) and 0.51 (Quant. 5). - Q1 values remain constant at -0.37 (Quant. 1-3) before dropping to -0.48 (Quant. 4) and -0.52 (Quant. 5). 2. **Color Consistency**: - Negative values (Q1) are consistently red/dark blue. - Positive values (Q3) transition from red (small positives) to green/orange (larger positives). 3. **Spatial Grounding**: - Both heatmaps share identical column labels but differ in value distributions. - Q3 values dominate the upper half (green/orange), while Q1 values occupy the lower half (red/dark blue). --- ### Interpretation 1. **Data Trends**: - The first heatmap suggests that quantization generally amplifies Q3 values while reducing Q1 values, with stabilization at Quant. 3-5. - The second heatmap shows a more nuanced pattern: Q3 values stabilize early (Quant. 1-3) before increasing, while Q1 values remain flat until later quantization levels. 2. **Outliers/Anomalies**: - In the first heatmap, Quant. 2 shows a slight divergence (Q3: 0.54, Q1: -0.53) compared to subsequent levels. - The second heatmap’s Quant. 5 exhibits the largest Q1 drop (-0.52), indicating potential sensitivity to extreme quantization. 3. **Technical Implications**: - Quantization appears to compress value distributions differently for Q1 and Q3, with Q3 benefiting more from higher quantization levels. - The divergence between heatmaps suggests methodological differences in quantization or data preprocessing. --- ### Conclusion The heatmaps reveal systematic shifts in quantile values across quantization levels, with Q3 consistently outperforming Q1 in magnitude. The second heatmap’s plateau and late-stage divergence highlight potential trade-offs in quantization strategies, warranting further investigation into optimization criteria. </details> Figure 2: Boxplot visualization of encoded latent Z reconstruction for fixed vs. adaptive quantization. Each subplot compares the distribution of an original latent Z against its cumulative reconstructions after successive quantization stages (’Quant. 1’-’Quant. 5’). The top panel depicts a fixed strategy using the first three quantizers, while the bottom panel shows an adaptive strategy selecting the three most suitable quantizers. ### 3.2 Residual Experts Vector Quantization Standard Residual Vector Quantization (RVQ) uses a fixed sequence of quantizers to process the latent representation. This approach is effective at high bitrates, but its performance degrades significantly at low bitrates, where it’s limited codebook struggles to represent diverse latent $Z$ , leading to substantial quantization error. Figure 2 illustrates this problem. A fixed strategy, using only the first few quantizers, results in a reconstruction that poorly matches the original latent distribution. In contrast, an adaptive strategy that selects the best-suited quantizers produces a far more accurate reconstruction. This clearly shows the need for a dynamic quantization method, which motivates our proposed Residual Experts Vector Quantization (REVQ). Our proposed REVQ implements this dynamic mechanism by adapting to the rich diversity found in audio signals over time. As visualized in Figure 1, the content of an audio waveform can change dramatically from one moment to the next. This temporal variability translates directly into a wide range of characteristics for the encoded latent $Z_e$ . A fixed set of quantizers, applied uniformly to all content, struggles to handle this diversity effectively. This allows us to treat each window as a distinct piece of content. For each window’s latent representation $Z_e$ , a router then selects the most suitable quantizer from a shared pool. By matching appropriate quantizers to the specific content, this approach significantly reduces quantization error. To assign routing quantizers for each audio segment, a gating network is employed, following the setup of DeepSeek-V3 [12], with a bias-free learnable matrix ${W^⊤}∈R^D× N_r$ used to compute affinity scores. Let $Z^\prime∈R^T× D$ denotes the transposed output of encoder, we compute affinity scores $S$ and $mask_i$ as follows: $$ S=\frac{1}{T}∑_t=1^T≤ft({Z^\prime}·{W^⊤}\right), \tag{1} $$ $$ mask_i=\begin{cases}1,&S_i∈Topk≤ft(\{S_j\mid 1≤ j≤ N_r\}, K_r\right),\\ 0,&otherwise,\end{cases} \tag{2} $$ where $N_r$ denotes the numbers of routed quantizers, $T$ denotes number of frames in the time domain after encoding; $D$ denotes the hidden dimension of encoded latent, Topk(S, K) denotes a function that selects the top $K$ largest values from $S$ . As shown in algorithm 1, the generated $mask_i$ is multiplied by the quantized output to select routed quantizers both in the encoder and decoder, which means that an additional bandwidth overhead of $N_r$ bits is introduced. The impact on bandwidth depends on the audio length. For example, a 2-second audio clip incurs an extra ( $\log_2N_r$ )/2 bps cost. Additionally, Table 4 demonstrates the performance improvement of our algorithm, showing that it maintains competitiveness with state-of-the-art codecs even in low-latency scenarios. Notably, a key design choice in our proposed REVQ is the decoupling of quantizer selection from their application order. While a subset of $K$ routing quantizers is adaptively chosen based on routing scores, their application follows a strict, predetermined sequence. Specifically, the chosen quantizers are applied not according to their selection scores, but in a fixed sequence determined by their original ascending indices. For instance, consider a scenario where quantizer 3 (with a higher affinity score) and quantizer 1 (with a lower affinity score) are both selected for a given input. Despite its lower score, quantizer 1 is invariably applied first to the initial residual. Subsequently, quantizer 3 is applied to process the new residual resulting from last quantization. This strict, index-based application order ensures that within any selected group, the lower-indexed quantizer consistently models the higher-energy components first. Consequently, REVQ inherits the principled, energy-descending hierarchy of traditional RVQ. This structured approach provides two key benefits: it enhances interpretability, as the routing mechanism learns to map high-energy latents to lower-indexed quantizers, and it improves training stability by assigning specialized, non-overlapping roles to each routing quantizer. This clear division of labor not only streamlines the model architecture but also reinforces robust learning. Since the mask involves in the quantization process is non-differentiable, we apply the straight-through estimator [39] to estimate the gradient for backpropagation as follows: $$ mask=S+sg(mask-S), \tag{3} $$ where sg denotes the stop-gradient operation. ### 3.3 Developing Router Protection Strategy Inadequate usage of routed quantizers, termed routing collapse [9], leads to wasted parameters and degraded performance. A mainstream approach [9] [10] [11] [40] to address this issue involves adding an auxiliary loss to the total model loss to penalize uneven expert allocation by the gate network. However, Wang et al. [41] demonstrated that while auxiliary losses enhance expert utilization, this approach often leads to model performance degradation. They propose an auxiliary-loss-free load balancing strategy by defining a bias matrix that adjusts affinity scores based on expert utilization, balancing token distribution without compromising model performance. All prior methods are based on large language models which own numerous gating networks. However, in our proposed REVQ framework, which employs a single lightweight matrix to assign quantizers, auxiliary losses and load-balancing learnable bias matrices trivially undermine our gating network. Specifically, these techniques force the network to output identical affinity scores for all routing quantizers, thereby disabling adaptive routing. Thus, we propose the Developing Router Protection Strategy, an approach inspired by Wang et al. ’s strategy, yet better tailored to our framework. Different from Wang et al., who reduce scores for quantizers above the average load and increase scores for those below it, we calculate gradient-free bias $b_i$ added to affinity scores $S_i$ as follows: $$ b_i=\begin{cases}b_i+γ,&load_i<active threshold,\\ 0,&load_i>\overline{load},\\ b_i,&otherwise,\end{cases} \tag{4} $$ $$ S_i=S_i+b_i, \tag{5} $$ where $γ$ denotes a hyperparameter that determines the intensity of the strategy intervention; $load_i$ denotes i-th routed quantizer usage accumulated over $n$ steps. For i-th routed quantizer whose $load_i$ is below the active threshold, $γ$ will be added to $b_i$ . For routed quantizers with a usage higher than the average, the bias added to them will be reset to zero. In simple terms, our goal is not to train mediocre, uniformly utilized routed quantizers but to create a competitive yet protective environment where quantizers are actively safeguarded from complete disuse while maintaining meaningful competition. <details> <summary>Figure_3.png Details</summary>  ### Visual Description ## Line Graphs: Frequency vs. Amplitude Analysis ### Overview The image contains three vertically stacked line graphs labeled (a) MRD, (b) First tier of MTSD, and (c) Second tier of MTSD. All graphs plot **Frequency (Hz)** on the x-axis (0–20,000 Hz) and **Amplitude** on the y-axis (-12.5 to +6). Each graph includes a legend at the bottom mapping colors to percentage ranges (0%-10%, 10%-25%, etc.). The graphs show distinct amplitude behaviors across frequency bands. --- ### Components/Axes 1. **X-Axis (Frequency)**: - Range: 0–20,000 Hz - Labels: 0, 2500, 5000, 7500, 10,000, 12,500, 15,000, 17,500, 20,000 Hz - Ticks: Every 2500 Hz (except MTSD graphs, which use 5000 Hz intervals). 2. **Y-Axis (Amplitude)**: - Range: -12.5 to +6 - Labels: -12.5, -10, -7.5, -5, -2.5, 0, 2.5, 5, 6 - Ticks: Every 2.5 units. 3. **Legends**: - Positioned at the bottom of each graph. - Color-coded percentage ranges: - Red: 0%-10% - Green: 10%-25% - Blue: 25%-50% - Yellow: 50%-75% - Pink: 75%-100% --- ### Detailed Analysis #### (a) MRD Graph - **Trend**: - **0%-10% (Red)**: Sharp peak at ~500 Hz (amplitude ~12), followed by rapid decay. - **10%-25% (Green)**: Small peak at ~2500 Hz (amplitude ~1). - **25%-50% (Blue)**: Flat line near 0 amplitude. - **50%-75% (Yellow)**: Flat line near 0 amplitude. - **75%-100% (Pink)**: Flat line near 0 amplitude. - **Key Data Points**: - Red peak: ~500 Hz, amplitude ~12. - Green peak: ~2500 Hz, amplitude ~1. #### (b) First Tier of MTSD - **Trend**: - **0%-10% (Red)**: Sharp peak at ~500 Hz (amplitude ~6), followed by decay. - **10%-100% (No color coding)**: Flat line near 0 amplitude. - **Key Data Points**: - Red peak: ~500 Hz, amplitude ~6. #### (c) Second Tier of MTSD - **Trend**: - **0%-10% (Blue)**: Sharp negative peak at ~500 Hz (amplitude ~-12), followed by decay. - **10%-100% (No color coding)**: Flat line near 0 amplitude. - **Key Data Points**: - Blue peak: ~500 Hz, amplitude ~-12. --- ### Key Observations 1. **Dominant Frequencies**: - All graphs show a dominant peak at ~500 Hz, suggesting a shared low-frequency resonance. - MRD (a) has the highest amplitude at this frequency (~12), while MTSD tiers show lower magnitudes (~6 and ~-12). 2. **Amplitude Decay**: - All graphs exhibit rapid amplitude decay beyond 500 Hz, with minimal activity above 2500 Hz. 3. **Negative Amplitude**: - Second tier of MTSD (c) shows a negative amplitude peak, potentially indicating phase inversion or opposing signal polarity. 4. **Color Consistency**: - Legends match graph colors exactly (e.g., red in MRD corresponds to 0%-10% frequency range). --- ### Interpretation - **MRD (a)**: The pronounced peak at 500 Hz (0%-10% range) suggests a strong resonant frequency in this band. The absence of activity in higher ranges (25%-100%) implies filtering or attenuation of higher frequencies. - **MTSD Tiers (b, c)**: The first tier (b) mirrors MRD’s 500 Hz peak but with reduced amplitude, possibly representing a secondary or attenuated signal. The second tier (c) introduces a negative amplitude peak, which could indicate a phase-shifted component or destructive interference. - **Technical Implications**: The graphs may represent spectral analysis of signals, with MRD showing raw data and MTSD tiers reflecting processed or filtered outputs. The negative amplitude in (c) warrants further investigation into signal polarity or phase relationships. *Note: All values are approximate due to lack of gridline precision.* </details> Figure 3: Comparison of STFT spectrogram segmentation strategies for discrimination. (a) The conventional Multi-Resolution Discriminator (MRD) employs a linear partitioning scheme, dividing the spectrogram into contiguous frequency sub-bands, with the lowest-frequency sub-band concentrating the vast majority of spectral information. (b, c) In contrast, our proposed Multi-Tiered STFT Discriminator (MTSD) utilizes a periodic, tiered partitioning strategy, resulting in sub-bands with a more balanced distribution of spectral information. <details> <summary>x2.png Details</summary>  ### Visual Description ## Diagram: Audio Signal Processing Neural Network Architecture ### Overview The diagram illustrates a convolutional neural network (CNN) architecture for audio signal processing. It shows the flow of data from raw waveform input through multiple processing stages to final logit outputs. The architecture combines time-frequency analysis with hierarchical convolutional feature extraction. ### Components/Axes 1. **Input Stage**: - **Waveform**: Raw audio input - **STFT (Short-Time Fourier Transform)**: Splits waveform into: - Imaginary Bins (orange circles) - Real Bins (blue squares) - **Time** (horizontal axis) and **Freq.** (vertical axis) dimensions 2. **Processing Blocks**: - **Concatenate**: Merges real/freq. and imaginary/time dimensions - **Multi-tiered**: Processes concatenated features - **Conv2D Blocks** (4 layers): - Channel=32 (gray) - Channel=64 (dark gray) - Channel=128 (medium gray) - Channel=256 (light gray) - **Conv2D Post**: Final processing layer 3. **Output**: - **Logits**: Single-channel output (blue oval) 4. **Legend**: - Orange: Imaginary components - Blue: Real components - Green: Time components - Yellow: Frequency components ### Detailed Analysis - **STFT Output**: - Imaginary bins: 4x4 grid (16 elements) - Real bins: 4x4 grid (16 elements) - Time dimension: 4 elements - Frequency dimension: 4 elements - **Conv2D Block Parameters**: - Channel sizes increase exponentially: 32 → 64 → 128 → 256 - Spatial dimensions reduce progressively through convolutional operations - **Color Coding**: - Orange circles consistently represent imaginary components - Blue squares consistently represent real components - Green squares represent time-domain features - Yellow squares represent frequency-domain features ### Key Observations 1. **Hierarchical Feature Extraction**: - Channel sizes double with each Conv2D layer (32→64→128→256) - Suggests multi-scale feature learning from simple to complex patterns 2. **Complex Number Handling**: - Separate processing of real/imaginary components from STFT - Recombination through concatenation before multi-tiered processing 3. **Temporal-Frequency Fusion**: - Time and frequency dimensions merged early in processing pipeline - Enables joint analysis of temporal and spectral features 4. **Output Structure**: - Final Conv2D Post layer reduces to single channel - Logits output suggests classification task (e.g., speech recognition) ### Interpretation This architecture demonstrates a sophisticated approach to audio processing by: 1. **Time-Frequency Decomposition**: Using STFT to capture both temporal and spectral characteristics 2. **Complex Feature Representation**: Maintaining phase information through separate real/imaginary processing 3. **Hierarchical Learning**: Progressively learning features at increasing abstraction levels through convolutional layers 4. **Efficient Output**: Final Conv2D Post layer condenses features to logits for classification The design suggests optimization for tasks requiring both temporal and spectral analysis, such as speech recognition or environmental sound classification. The exponential growth in channel sizes indicates a focus on capturing increasingly complex audio patterns through deeper network layers. </details> Figure 4: Architecture of the Multi-Tiered STFT Discriminator (MTSD). The discriminator takes an input waveform and first computes its STFT, separating it into real (squares) and imaginary (circles) bins. These components are then concatenated along the time dimension and fed into the core Multi-Tiered module. This module repartitions the data into several parallel tiers, each containing a balanced mix of frequency bins from across the spectrum. These tiers are processed by a stack of 2D convolutional blocks, which extract hierarchical features with progressively increasing channel depth (32 to 256). Finally, the discriminator generates multi-scale outputs: individual logits are extracted from the feature maps of each convolutional block, and these are combined with the output of the final post-convolutional layer to form the overall judgment. ### 3.4 Multi-Tiered STFT Discriminator For generative audio models that employ adversarial training, the discriminator is the key component that guides the generator toward high-fidelity synthesis. A prevalent approach is the Multi-Resolution Discriminator (MRD) as described in section 2.3, which evaluates audio quality by decomposing the spectrogram into distinct sub-bands for parallel assessment. However, we argue that its linear partitioning strategy is suboptimal, as it often creates a severe imbalance of information density across these sub-bands. To illustrate this imbalance, Figure 3 (a) displays that MRD’s linear partitioning scheme concentrates nearly all spectral energy into the lowest sub-band. This severe informational imbalance directly compromises the discriminator’s effectiveness. Since the discriminator typically assigns equal weight to the score of each sub-band, the generator can achieve a high overall score simply by perfectly reconstructing the high-frequency bands, which contain little complex information. This low-frequency band is the most critical, as it is information-dense and where most reconstruction errors are likely to occur. However, its contribution to the overall score is outweighed by the combined scores of the many easily matched high-frequency bands. As a result, the discriminator is misled by high scores from these less important regions and fails to provide meaningful feedback on the most critical parts of the spectrum. To overcome this limitation, we introduce the Multi-Tiered STFT Discriminator (MTSD). Our approach re-engineers the frequency decomposition by partitioning the spectrogram into tiers with comparable information density. As illustrated in sub-figures (b) and (c), we employ a periodic partitioning scheme to create a rich and balanced set of spectral components from across the entire frequency range. This design ensures that each sub-discriminator receives a meaningful and information-dense input, thereby enhancing its ability to distinguish between real and generated spectra. This, in turn, provides the generator with more targeted and effective feedback, compelling it to refine subtle details and produce a higher-fidelity spectrum. To achieve this, we now detail the architecture of our MTSD. The MTSD consists of three sub-discriminators that operate at different scales, each of which accepts only equally spaced frequency bins at same resolution. The input audio undergoes different STFTs to obtain 256, 512, and 1024 frequency bins per frame respectively. To facilitate phase modeling and preserve periodicity, we employ a multi-tier partitioning scheme, which is visually depicted in Figure 4. The process begins by concatenating the real part and imaginary part in the time domain. Subsequently, the resulting frequency bins $f$ are partitioned periodically into $p$ tiers, each with a length of $f/p$ . We set the periods $p$ to [2, 4, 8] to unify the resolutions and architecture among the sub - discriminators. The backbone is composed of a sequence of blocks, where each block consists of a 2D convolution layer with a kernel size of 3 $×$ 9 and a stride of 1 $×$ 2, followed by a LeakyReLU activation layer with a negative slope of 0.1. The number of channel is first halved from 128 to 32 in the first convolutional layer, then progressively doubled up to 256 as the network depth increases. Finally, a downsampling convolutional layer is employed to reduce the number of channels to 1. Each hidden feature after convolution is retained as logits for MTSD. ### 3.5 Quantizer Dropout With Post-Training When the encoder structure of the model and the codebook size are fixed, the bitrate is determined by the number of codebooks used. SoundStream [3] introduces Quantizer dropout into RVQ to ensure the model is able to adjust to varying bitrates by randomly sampling the number of codebooks during training. To adapt quantizer dropout to our architecture, we select the top- $k$ quantizers based on their affinity scores, where the value of $k$ for each training instance is stochastically sampled from the range [1, $N_r$ ]. In our REVQ framework, codebooks within quantizers are updated via gradient, meaning that quantizers selected more frequently receive more optimization steps. As training progresses, this leads to a selection bias where the active quantizer subset stabilizes far below the theoretical maximum. By applying dropout to our model, we increase selection diversity among routing quantizers, mitigating the ”winner-takes-all” observed in configurations where a fixed number of quantizers are used per inference. Notably, while dropout enables variable bitrate support, it often compromises performance, especially at lower bitrates. To counteract this, we introduce a simple yet effective post-training strategy that achieves comparable performance without the cost of full retraining. Our method begins with a pre-trained model with droput and fine-tunes it for a few additional epochs with the model locked to a specific target bitrate. This brief period of targeted adaptation allows the model to rapidly optimize its parameters for the fixed bitrates condition. Consequently, our post-training strategy drastically reduces the cumulative computational cost required to support multiple bitrates. This efficiency is achieved by deriving high-performance, specialized models from a single pre-trained foundation, rather than training each one individually from scratch. <details> <summary>Figure_5.png Details</summary>  ### Visual Description ## Heatmap: Frequency-Time Analysis Across Signal Processing Methods ### Overview The image presents a comparative heatmap analysis of frequency distributions over time for four signal processing scenarios: Natural, SwitchCodec, DAC, and Encodec. Each panel uses a color gradient (dark purple to yellow) to represent intensity, with a blue box highlighting a specific time-frequency region (0.1-0.4s, 5000-7500Hz) for detailed comparison. ### Components/Axes - **X-axis (Time)**: 0.000–0.400 seconds, linear scale - **Y-axis (Frequency)**: 0–20,000 Hz, linear scale - **Color Scale**: Dark purple (low intensity) to yellow (high intensity) - **Panels**: - (a) Natural: Baseline signal - (b) SwitchCodec: Compressed signal - (c) DAC: Decompressed signal - (d) Encodec: Hybrid encoding/decoding - **Inset**: Zoomed view of the blue-boxed region (0.1–0.4s, 5000-7500Hz) ### Detailed Analysis 1. **Natural (a)**: - Smooth gradient from dark purple (low) to yellow (high) across all frequencies. - No abrupt transitions; represents unprocessed signal characteristics. 2. **SwitchCodec (b)**: - Blocky patterns in mid-frequency ranges (5000–15,000Hz). - Reduced intensity in the highlighted region compared to Natural. - Inset shows fragmented yellow patches, suggesting compression artifacts. 3. **DAC (c)**: - Uniform yellow distribution in the highlighted region. - Smoother transitions than SwitchCodec but less detailed than Natural. - Inset reveals consistent intensity across the zoomed region. 4. **Encodec (d)**: - Hybrid pattern: Blocky mid-frequencies with smoother high/low ranges. - Highlighted region shows mixed intensity (yellow/purple), indicating partial reconstruction. - Inset displays granular yellow areas, suggesting residual artifacts. ### Key Observations - **Artifact Patterns**: SwitchCodec introduces blocky distortions in mid-frequencies, while Encodec shows a mix of artifacts and smoother regions. - **Frequency Preservation**: Natural retains full frequency detail; DAC partially restores it, and Encodec shows intermediate recovery. - **Time-Frequency Localization**: The blue-boxed region (0.1–0.4s, 5000-7500Hz) is critical for comparing compression/decompression fidelity. ### Interpretation The data demonstrates how encoding methods alter frequency-time relationships: - **Natural** serves as a reference for unprocessed signal integrity. - **SwitchCodec** prioritizes compression efficiency at the cost of mid-frequency detail, evident in the blocky patterns. - **DAC** partially restores the signal but lacks the granularity of the original. - **Encodec** balances compression and reconstruction, retaining some mid-frequency structure while smoothing extremes. The highlighted regions emphasize trade-offs: SwitchCodec sacrifices detail for efficiency, DAC recovers broadly but imprecisely, and Encodec offers a middle ground. These patterns align with expectations for lossy compression artifacts and hybrid encoding strategies. </details> Figure 5: Comparison of mel spectrograms: (a) natural mel spectrogram; (b), (c), (d) mel spectrograms generated by SwitchCodec, DAC, and EnCodec, respectively. ## 4 Experiment ### 4.1 Dataset We train our model on large-scale datasets spanning speech and music domains. For Speech, we use the VCTK dataset [47] from University of Edinburgh, contains 110 diverse speakers with multi-accent English recordings; the train-clean-100 LibriTTS dataset [48], derived from the LibriSpeech ASR corpus, offers 100 hours clean read English speech from multi-speakers; the Common Voice dataset [49] created by Mozilla includes global linguistic diversity. For music, we rely on the FMA dataset [50], which provides high-quality audio across 161 genres and serves as a foundation for multi-genre music modeling. All audio is resampled to 44.1 kHz, with FMA music tracks converted to mono-channel for input dimension standardization. For evaluation, we select 360 audio clips from the test sets of VCTK, LibriTTS, Common Voice, and FMA to ensure cross-domain generalization validation. ### 4.2 Experimental Setup Our model employs convolutional encoder-decoder networks to extract high-dimensional audio features, which builds upon DAC [24]. These features undergo compression through Residual Experts Vector Quantization (REVQ) with one shared quantizer and a few quantizers sparsely activated from eight available quantizers. Our model use Multi-Period Discriminator (MPD) [42] and Multi-Tiered STFT Discriminator (MTSD) to enhance waveform and spectral reconstruction through adversarial loss. We utilize four NVIDIA RTX 4080 GPUs for experiments. Models for ablation study are trained with a batch size of 8 for 100k iterations on a single GPU. For our final model, we train with a batchsize of 32 for 400k iterations, using peroids of [2,3,5,7,11] for MPD and [8,4,2] for MTSD. We adopte Developing Router Protection Strategy (DRPS) whose $γ$ is set to 0.01 for quantizers utilization improvement. We apply a modified quantizer dropout technique to enable our model to support nine bitrates ranging from 0.89 kbps to 8 kbps. Our model accepts audio excerpts of 0.38s in duration for training. During inference, we split audio into segments with a window size of 1s, individually feed them into the model, and assign customized routing quantizers to each segment. Following baseline, we use the AdamW optimizer [43] with a learning rate of $1e-4$ , $β_1=0.8$ , and $β_2=0.9$ , for both the generator and the discriminator, and decay the learning rate at every step, with a factor of 0.999996. ### 4.3 Metric For subjective evaluation, we conduct a listening test following a methodology inspired by the MUSHRA [44]. In this test, our proposed codec at 2.67 kbps and 5.33 kbps are compared against the baseline model. A total of 20 experienced listeners, equipped with headphones, participated in a double-blind evaluation. Each of the randomly selected audio samples is presented with a hidden reference, but without a low-pass filtered anchor. To ensure the reliability of the results, we implement a screening process: ratings from participants who scored the hidden reference below 90 or any other sample above the reference are discarded. For objective evaluation, we utilize four established metrics: Perceptual Evaluation of Speech Quality (PESQ) [45], Virtual Speech Quality Objective Listener (ViSQOL) [46], Mel Distance, and STFT Distance. - PESQ: Quantifies speech quality by comparing a reference signal to a degraded one through a psychoacoustic model. - ViSQOL: Measures the spectro-temporal similarity between reference and test signals using a model of human auditory perception, making it effective for a broad range of audio quality assessments. - Mel Distance: Assesses perceptual fidelity by computing the L1 distance over multi-scale mel spectrograms, which are generated using multiple resolutions (window lengths: 32-2048 samples). - STFT Distance: Evaluates spectral fidelity through a composite score, combining the L1 distance on both the linear STFT magnitudes and their logarithmic counterparts. Table 1: Objective evaluation of the proposed codec at varying bitrates, along with results from competing approaches. | Neural Audio Codec | Bitrate (kbps) | Bandwidth (kHz) | Mel distance $↓$ | STFT distance $↓$ | PESQ $↑$ | ViSQOL $↑$ | | --- | --- | --- | --- | --- | --- | --- | | SwitchCodec | 2.67 | 44.1 | 0.75 | 1.71 | 2.87 | 4.04 | | 5.33 | 44.1 | 0.66 | 1.65 | 3.49 | 4.25 | | | EnCodec | 3 | 48 | 1.20 | 2.43 | 1.71 | 2.09 | | 6 | 48 | 1.06 | 2.29 | 2.21 | 2.71 | | | 12 | 48 | 0.94 | 2.19 | 2.76 | 3.36 | | | DAC | 2.67 | 44.1 | 0.87 | 1.89 | 2.31 | 3.61 | | 3.56 | 44.1 | 0.81 | 1.83 | 2.72 | 3.72 | | | 4.44 | 44.1 | 0.76 | 1.80 | 3.05 | 3.81 | | | 5.33 | 44.1 | 0.72 | 1.77 | 3.31 | 3.87 | | | 6.22 | 44.1 | 0.68 | 1.74 | 3.52 | 3.92 | | <details> <summary>Figure_6.png Details</summary>  ### Visual Description ## Scatter Plot: MUSHRA Score vs Bitrate (kbps) ### Overview The image displays a scatter plot comparing audio codec performance using MUSHRA (Multimedia User Survey for Hearing-Related Assessment) scores against bitrates (kbps). Three codecs are compared: SwitchCodec (red stars), DAC (orange circles), and EnCodec (green triangles). A reference line at 95 MUSHRA score is shown in blue. ### Components/Axes - **X-axis**: Bitrate (kbps) ranging from 2.67 to 6.0 kbps in increments of ~0.33 kbps. - **Y-axis**: MUSHRA Score (40–100) with gridlines at 10-point intervals. - **Legend**: Located at the top-right corner, associating colors with codecs: - Red: SwitchCodec - Orange: DAC - Green: EnCodec - **Reference Line**: Blue dashed horizontal line at 95 MUSHRA score. ### Detailed Analysis 1. **SwitchCodec (Red Stars)**: - Data points at: - (2.67 kbps, 90) - (5.33 kbps, 93) - Trend: Scores increase slightly with bitrate, approaching the reference line. 2. **DAC (Orange Circles)**: - Data points at: - (2.67 kbps, 87) - (3.56 kbps, 88) - (5.33 kbps, 89) - Trend: Gradual improvement with bitrate, but scores remain below SwitchCodec. 3. **EnCodec (Green Triangles)**: - Data points at: - (3.0 kbps, 68) - (6.0 kbps, 70) - Trend: Minimal improvement despite doubling the bitrate. ### Key Observations - **Performance Hierarchy**: SwitchCodec consistently outperforms DAC and EnCodec across all bitrates. - **Reference Line Gap**: No codec reaches the 95 MUSHRA score target, though SwitchCodec at 5.33 kbps (93) is closest. - **EnCodec Anomaly**: Despite the highest bitrate (6.0 kbps), EnCodec scores remain significantly lower than DAC at lower bitrates (e.g., 3.56 kbps: 88 vs. 70). ### Interpretation The data suggests that **SwitchCodec** achieves the highest perceptual quality (MUSHRA score) relative to bitrate, making it the most efficient codec. DAC performs mid-tier, while EnCodec lags despite higher bitrates, indicating potential inefficiencies in its encoding/decoding process. The reference line at 95 implies an aspirational quality threshold, with SwitchCodec being the only codec to approach it. The trend of increasing scores with bitrate aligns with expectations, but EnCodec’s underperformance raises questions about its technical implementation or perceptual optimization. </details> Figure 6: Subjective listening tests for SwitchCodec, DAC, EnCodec and the reference across various bitrates. Table 2: Ablation study for our proposed SwitchCodec. | MTSD | REVQ | Gamma | PESQ $↑$ | Mel Dist. $↓$ | STFT Dist. $↓$ | ViSQOL $↑$ | | --- | --- | --- | --- | --- | --- | --- | | ✗ | ✗ | ✗ | $2.26$ | $0.99$ | $1.97$ | $3.71$ | | [32,16,8] | ✗ | ✗ | $2.18$ | $0.91$ | $1.89$ | $3.80$ | | [16,8,4] | ✗ | ✗ | $2.30$ | $0.90$ | $1.89$ | $3.83$ | | [8,4,2] | ✗ | ✗ | $2.36$ | $0.90$ | $1.96$ | $3.86$ | | [8,4,2] | ✓ | ✗ | $2.57$ | $0.82$ | $1.79$ | $3.92$ | | [8,4,2] | ✓ | 0.1 | $2.45$ | $0.84$ | $1.80$ | $3.82$ | | [8,4,2] | ✓ | 0.01 | 2.55 | 0.82 | 1.68 | 4.07 | | [8,4,2] | ✓ | 0.001 | $2.42$ | $0.84$ | $1.81$ | $3.90$ | - The top row shows the objective scores of the official DAC model, trained with the author-provided configuration. ### 4.4 Comparison of models We now compare the performance of our final model with competitive baselines: DAC [24], EnCodec [22]. For DAC and EnCodec, we utilize the pre-trained models provided by the authors. We present mel spectrogram comparisons with baselines in Figure 5. It can be observed that our model significantly reduces spectral blurring and generates high-fidelity results. We compare all the codecs using both objective and subjective evaluations at varying bitrates. The results in Table 1 and Figure 6 shows that our proposed codec outperforms all competing codecs even at higher compression in terms of both objective and subjective metrics. An online demo is available at: https://raconiy.github.io/Switchcodec/index.html. ### 4.5 Ablation study We conduct a comprehensive ablation study of our model, removing and modifying individual components of our training protocol to demonstrate their impact. To compare models, we use the four objective metrics described in Section 4.2. The results of our ablation study can be seen in Table 1. #### 4.5.1 Discriminator design To optimize our proposed MTSD and demonstrate its efficacy, we perform an ablation study on its core hyperparameter: the number of stratification tiers. We then compared our optimized configuration against the baseline model. We begin our ablation study by examining the impact of the number of stratification tiers in the MTSD, as detailed in Table 2. The results reveal a clear performance trade-off. Decreasing the number of tiers from a configuration of [32, 16, 8] to [8, 4, 2] yields a consistent improvement in perceptual quality. Specifically, the PESQ score rises from 2.18 to 2.36, and ViSQOL increases from 3.80 to 3.86. This trend suggests that a finer-grained stratification encourages the generator to better preserve features salient to human auditory perception. Conversely, this enhancement in perceptual metrics is accompanied by a slight increase in the STFT distance (from 1.89 to 1.96). Given that the primary target of a high-fidelity codec is to optimize for perceptual experience, we identify the [8, 4, 2] configuration, which employs the least number of stratification tiers, as the optimal design for our model. Having identified [8, 4, 2] as the optimal setting, we highlight the significant performance gains achieved by replacing the baseline MRD with our proposed MTSD. Our model, when equipped with the MTSD, surpasses the MRD-equipped baseline across all metrics: PESQ increased from 2.26 to 2.36, and ViSQOL is lifted substantially from 3.71 to 3.86. Furthermore, our MTSD also improves signal reconstruction fidelity, reducing the mel-distance from 0.99 to 0.90 and the STFT-distance from 1.97 to 1.96. These results validate that the MTSD is more effective than the MRD, yielding considerable improvements in both objective perceptual quality and spectrogram accuracy. Therefore, all subsequent experiments adopt the MTSD with the [8, 4, 2] configuration. Table 3: Study on the impact of the number of available quantizers on audio quality and their actual usage during inference without DRPS. | N_Quant. | REVQ | | | | | --- | --- | --- | --- | --- | | PESQ | Mel Loss | VISQOL | Usage | | | 5 | 2.53 | 0.83 | 3.92 | 100.0% | | 7 | 2.53 | 0.82 | 3.89 | 71.4% | | 9 | 2.57 | 0.82 | 3.94 | 44.4% | | 17 | 2.57 | 0.81 | 3.92 | 16.6% | #### 4.5.2 Sparse Quantization Different from the baseline, which uses only three quantizers at 2.67 kbps, REVQ sparsely activate three quantizers from $N_q$ quantizers, resulting in a $(N_q/3)×$ larger embedding space. During inference, one quantizer serves as a shared quantizer, and two act as routed quantizers, representing $C_N_{q-1}^2$ possible selection combinations, whereas the baseline has only one fixed selection. To demonstrate the advantages of our proposed REVQ, we benchmark it against a standard Residual Vector Quantization (RVQ) implementation. As shown by the comparison in Table 2, integrating REVQ (configured with 8 available quantizers) into our model yields substantial performance improvements. Specifically, this substitution significantly increases the PESQ score from 2.36 to 2.57 and the ViSQOL score from 3.86 to 3.92. Concurrently, reconstruction fidelity is enhanced, with mel-distance and STFT-distance decreasing from 0.90 and 1.96 to 0.82 and 1.79, respectively. These results unequivocally establish the benefits of our sparse quantization approach. While REVQ proves effective, its performance does not scale linearly with the number of available quantizers. To investigate this, we varies the size of the quantizer pool, with results presented in Table 3. Our findings reveal a critical limitation of the routing mechanism: as the pool expands, the performance gains quickly diminish, plateauing around a PESQ score of 2.57. For instance, increasing the number of routing quantizers from 5 to 17 yields no meaningful improvement. The underlying cause for this performance saturation is a drastic drop in quantizer utilization. While a pool of 5 quantizers is fully utilized (100%), the usage rate plummets to just 44.4% for 9 available quantizers and a mere 16.6% for 17. This indicates that the routing mechanism fails to effectively leverage the expanded selection, leaving most quantizers neglected. This severe underutilization bottlenecks the model’s potential, motivating the need for an improved mechanism to enhance expert engagement, which we introduce in the following section. #### 4.5.3 Routed Quantizers Load Balance To address the quantizer underutilization identified previously, we introduce the Developing Router Protection Strategy (DRPS), detailed in Section 3.3. We conduct an ablation study on its core hyperparameter, $γ$ , which governs the strength of the corrective bias applied to inactive quantizers. The results, presented in Table 2 and Figure 7, underscore the critical role of $γ$ in achieving optimal performance. Our analysis reveals that both excessive and insufficient values for $γ$ are counterproductive. An overly aggressive setting (e.g., $γ$ =0.1) proved detrimental, degrading performance below the baseline without DRPS. As shown in Figure 7, this high value paradoxically worsen quantizer collapse by reducing the number of activated quantizers (NAQ) from 3 to 2. Conversely, a minimal value (e.g., $γ$ =0.001) is insufficient to fully counteract underutilization. While it moderately increases NAQ to 6, this is not enough to yield a net performance benefit, with key perceptual metrics remaining below the baseline. In contrast, an intermediate value of $γ$ =0.01 strikes the optimal balance. <details> <summary>Figure_7.png Details</summary>  ### Visual Description ## Combination Chart: NAQ and ViSQOL Metrics vs. γ Parameter ### Overview The chart compares two metrics (NAQ and ViSQOL) across four γ parameter values (0, 0.1, 0.01, 0.001). NAQ is represented by peach-colored bars, while ViSQOL is shown as a black line with square markers. The chart reveals inverse relationships between γ and the metrics at certain thresholds. ### Components/Axes - **X-axis (γ)**: Discrete values [0, 0.1, 0.01, 0.001] (logarithmic spacing) - **Left Y-axis (NAQ)**: Scale 0–8 (integer increments) - **Right Y-axis (ViSQOL)**: Scale 3.80–4.15 (0.05 increments) - **Legend**: - NAQ: Peach-colored bars (top-left position) - ViSQOL: Black line with square markers (top-left position) ### Detailed Analysis **NAQ Bars**: - γ=0: ~3.0 - γ=0.1: ~2.0 - γ=0.01: ~7.0 (peak) - γ=0.001: ~6.0 **ViSQOL Line**: - γ=0: 3.90 - γ=0.1: 3.85 - γ=0.01: 4.10 (peak) - γ=0.001: 3.95 ### Key Observations 1. **Peak Alignment**: Both metrics reach maximum values at γ=0.01 (NAQ=7.0, ViSQOL=4.10) 2. **Inverse Relationships**: - At γ=0.1: NAQ (2.0) < ViSQOL (3.85) - At γ=0.001: NAQ (6.0) > ViSQOL (3.95) 3. **Divergence at γ=0**: NAQ (3.0) > ViSQOL (3.90) despite lower γ value 4. **Non-linear Scaling**: ViSQOL shows smaller absolute changes (0.25 max) vs. NAQ (4.0 max) ### Interpretation The data suggests γ=0.01 represents an optimal threshold where both metrics achieve their highest values. Below this threshold (γ=0.001), NAQ decreases more significantly than ViSQOL, indicating potential trade-offs between the metrics at extreme γ values. The inverse relationship at γ=0.1 suggests γ magnitude critically influences metric performance. This pattern could reflect a system where intermediate γ values balance competing objectives represented by NAQ and ViSQOL. </details> Figure 7: Effect of DRPS on quantizers usage and audio quality during inference, NAQ: Number of Activated Quantizers. This setting successfully engages all 8 available quantizers during inference (Figure 7), completely resolving the underutilization issue. This full engagement translates directly into superior audio quality, as evidenced in Table 2. While the PESQ score (2.55) remains comparable to the baseline, the ViSQOL score significantly improves to 4.07, and the STFT-distance reaches a new minimum of 1.68. This indicates a tangible enhancement in both perceptual quality and signal fidelity. Based on this comprehensive analysis, we adopt $γ$ =0.01 for all subsequent experiments. ### 4.6 Quantizer Dropout With Post-Training To evaluate the side-effects of dropout on the model and the mitigating impact of post-training, we pre-train models with dropout for 100k and 200k iterations. Additionally, we train a model for 200k iteration at multiple specific bitrates to serve as a baseline for comparison. Starting from the model with dropout pretrained for 100k iterations, we conduct additional 100k-iteration fine-tuning at each target bitrate. This ensures all evaluated models undergo same training iterations. <details> <summary>Figure_8.png Details</summary>  ### Visual Description ## Line Chart: PESQ vs Bitrate (bps) ### Overview The chart compares three methods (dropout, w/o dropout, post-training) across bitrates (3-5 bps) using the PESQ metric. All three methods show decreasing PESQ scores as bitrate decreases, with post-training consistently outperforming the others. ### Components/Axes - **Y-axis**: PESQ (2.4–3.4) - **X-axis**: Bitrate (bps) (3–5) - **Legend**: Top-right corner with three entries: - Black dashed line + squares: dropout - Red dashed line + circles: w/o dropout - Blue dashed line + triangles: post-training ### Detailed Analysis 1. **Dropout (Black)**: - At 5 bps: ~3.25 PESQ - At 4 bps: ~3.0 PESQ - At 3 bps: ~2.5 PESQ - Trend: Steepest decline, dropping 0.75 PESQ across bitrates. 2. **w/o Dropout (Red)**: - At 5 bps: ~3.3 PESQ - At 4 bps: ~3.1 PESQ - At 3 bps: ~2.65 PESQ - Trend: Gradual decline, dropping 0.65 PESQ across bitrates. 3. **Post-Training (Blue)**: - At 5 bps: ~3.2 PESQ - At 4 bps: ~3.05 PESQ - At 3 bps: ~2.6 PESQ - Trend: Moderate decline, dropping 0.6 PESQ across bitrates. ### Key Observations - Post-training maintains the highest PESQ scores at all bitrates. - Dropout method shows the most significant performance degradation as bitrate decreases. - w/o dropout performs better than dropout but worse than post-training. ### Interpretation The data suggests that **post-training** is the most effective method for preserving audio quality (as measured by PESQ) across varying bitrates. The dropout technique appears to introduce the greatest performance penalty, likely due to its inherent noise-reduction mechanism. The w/o dropout baseline demonstrates that removing dropout improves performance but does not match post-training's effectiveness. This implies that post-training optimizes the model's ability to handle low-bitrate audio without sacrificing quality, making it ideal for bandwidth-constrained applications. </details> Figure 8: PESQ scores for the model using dropout, models trained with different bitrates, and the model employing post-training. Figure 8 demonstrates that quantizer dropout compromises model performance, exhibiting more severe degradation at lower bitrates. However, models with post-training close the gap with those without dropout, with their differences reduced, especially at lower bitrates. Our post-training approach achieves comparable model performance while halving the training time. Table 4: Evaluation under streamable and non-streamable inference of DAC and SwitchCodec. | Model | PESQ | ViSQOL | | --- | --- | --- | | Non-Streamable | | | | DAC (3.56 kbps) | 2.72 | 3.72 | | SwitchCodec (2.67 kbps) | 2.87 | 4.04 | | Streamable | | | | DAC (3.56 kbps) | 1.12 | 1.06 | | SwitchCodec (2.82 kbps) | 1.32 | 2.80 | ### 4.7 Low-Latency Inference As mentioned in Section 3.2, our approach incurs additional bandwidth costs proportional to audio length. In non-streaming scenarios, this bandwidth overhead is negligible, but in low-latency contexts, where audio is framed into millisecond-scale lengths, its side effects become significant. To assess the performance of our model under low-latency conditions, we conduct a evaluation with a 20ms frame size, comparing our proposed SwitchCodec against the DAC baseline. The experimental settings for both non-streaming and streaming scenarios are detailed below. In the non-streamable inference, SwitchCodec utilizes 3 quantizers, operating at 2.67 kbps, while the DAC [24] baseline uses 4 quantizers at 3.56 kbps. For streamable inference, however, SwitchCodec incurs a minor bandwidth overhead. It must transmit a mask of $N_r$ bits per frame to inform the decoder which routing quantizers are selected. In this experiment, with $N_r=2^3$ and a 20ms frame size (50 frames/sec), this results in an additional bandwidth cost of 150 bps. Consequently, the actual bitrate of SwitchCodec in the streaming scenario increases to 2.82 kbps. As presented in Table 4, our findings demonstrate the superiority of SwitchCodec across all conditions. In both streamable and non-streamable inference, SwitchCodec consistently outperforms the state-of-the-art DAC model, all while operating at a lower bitrate. We demonstrate that our method remains competitive even in low-latency inference. ## 5 Conclusion In this paper, we have presented a novel neural audio codec that achieves impressive compression while maintaining excellent audio fidelity, even under low-latency constraints. Our core contribution is the introduction of a sparsity-driven quantization mechanism, which dynamically adapts to diverse audio characteristics and significantly reduces quantization error. Furthermore, we introduce two key enhancements: a strategy to ensure the full utilization of the available quantizers and an improved discriminator architecture for more accurate spectrogram modeling. We also proposed a generalizable post-training strategy to counteract quality degradation from quantizer dropout, a technique applicable to the broader family of RVQ-based models. For future work, we will explore the application of our method to large language models and target further advancements in extremely low-bitrate audio compression. ## References - [1] A. Van Den Oord, and O. Vinyals, Neural discrete representation learning, Advances in Neural Information Processing Systems. 30 (2017). - [2] Y.-W. Guo et al., Recent Advances in Discrete Speech Tokens: A Review, 2025, arXiv preprint arXiv:2502.06490. - [3] N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi, SoundStream: An End-to-End Neural Audio Codec, IEEE/ACM Trans. on Audio, Speech, and Language Processing. 30 (2022) 495-507. - [4] B.-H. Juang, and A. Gray, Multiple Stage Vector Quantization for Speech Coding, in: Proc. IEEE ICASSP, 1982, pp. 597-600. - [5] D. Yang et al., HiFi-Codec: Group-Residual Vector Quantization for High Fidelity Audio Codec, 2023, arXiv preprint arXiv:2305.02765. - [6] Y. Chae et al., Variable Bitrate Residual Vector Quantization for Audio Coding, in: Proc. IEEE ICASSP, 2025, pp. 1-5. - [7] J. Pons, S. Pascual, G. Cengarle, and J. Serrà, Upsampling artifacts in neural audio synthesis, in: ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, 2021, pp. 3005-3009. - [8] M. Morrison, R. Kumar, K. Kumar, P. Seetharaman, A. Courville, and Y. Bengio, Chunked autoregressive gan for conditional waveform synthesis, 2021, arXiv preprint arXiv:2110.10139. - [9] N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton and J. Dean, Outrageously large neural networks: The sparsely-gated mixture-of-experts layer, 2017, arXiv preprint arXiv:1701.06538. - [10] D. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen, Gshard: Scaling giant models with conditional computation and automatic sharding, International Conference on Learning Representations. (2021). - [11] W. Fedus, B. Zoph, and N. Shazeer, Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, Journal of Machine Learning Research. 23 (2022). - [12] A. Liu et al., Deepseek-v3 technical report, 2024, arXiv preprint arXiv:2412.19437. - [13] J.-M. Valin, K. Vos, and T. B. Terriberry, Definition of the opus audio codec, IETF RFC 6716, 2012, [Online]. Available: https://tools.ietf.org/ html/rfc6716 - [14] M. Dietz et al., Overview of the EVS codec architecture, in: Proc. IEEE Int. Conf. Acoust., Speech, Signal Process, 2015, pp. 5698-5702. - [15] M. Neuendorf et al., The ISO/MPEG Unified speech and audio coding standard - Consistent high quality for all content types and at all bit rates, J. Audio Eng. Soc. 61 (2013) 956-977. - [16] C. Gârbacea et al., Low bit-rate speech coding with VQ-VAE and a WaveNet decoder, in: Proc. IEEE Int. Conf. Acoust., Speech, Signal Process, 2019, pp. 735-739. - [17] A. van den Oord et al., WaveNet: A generative model for raw audio, 2016, arXiv preprint arXiv:1609.03499. - [18] J. Eric, S. Gu, and B. Poole, Categorical reparameterization with gumbel-softmax, 2016, arXiv preprint arXiv:1611.01144. - [19] X. Jiang, X. Peng, H. Xue, Y. Zhang and Y. Lu, Latent-Domain Predictive Neural Speech Coding, IEEE/ACM Trans. on Audio, Speech, and Language Processing, IEEE, 2023, vol. 31, pp. 2111-2123, doi: 10.1109/TASLP.2023.3277693. - [20] X. Jiang, X. Peng, Y. Zhang, Y. Lu, Disentangled feature learning for real-time neural speech coding, in: ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, 2023, pp. 1-5. - [21] Y. Li, M. Tagliasacchi, O. Rybakov, V. Ungureanu, and D. Roblek, Realtime speech frequency bandwidth extension, in: Proc. IEEE Int. Conf. Acoust., Speech, Signal Process, 2021, pp. 691-695. - [22] A. Défossez, J. Copet, G. Synnaeve, and Y. Adi, High fidelity neural audio compression, Transactions on Machine Learning Research. (2023). - [23] S. Lee, W. Ping, B. Ginsburg, B. Catanzaro, and S. Yoon, Bigvgan: A universal neural vocoder with large-scale training, 2022, arXiv preprint arXiv:2206.04658. - [24] R. Kumar, P. Seetharaman, A. Luebs, I. Kumar, and K. Kumar, High-fidelity audio compression with improved rvqgan, Advances in Neural Information Processing Systems. 36 (2023). - [25] Z. Liu, T. Hartwig, and M. Ueda. Neural networks fail to learn periodic functions and how to fix it, Advances in Neural Information Processing Systems. 33 (2020) 1583-1594. - [26] J. Yu, X. Li, J. Y. Koh, H. Zhang, R. Pang, J. Qin, A. Ku, Y. Xu, J. Baldridge, and Y. Wu. Vector-quantized image modeling with improved vqgan, 2021, arXiv preprint arXiv:2110.04627. - [27] X. Bie, X. Liu, G. Richard, Learning Source Disentanglement in Neural Audio Codec, 2024, arXiv preprint arXiv:2409.11228. - [28] Z. Borsos, M. Sharifi, D. Vincent, E. Kharitonov, N. Zeghidour, M. Tagliasacchi, Soundstorm: Efficient parallel audio generation, 2023, arXiv preprint arXiv:2305.09636. - [29] H. Li, L. Xue, H. Guo, X. Zhu, Y. Lv, L. Xie, Y. Chen, H. Yin, and Z. Li, Single-codec: Single-codebook speech codec towards high-performance speech generation, 2024, arXiv preprint arXiv:2406.07422. - [30] S. Ji et al., WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling, 2024, arXiv preprint arXiv:2408.16532. - [31] A. Vasuki, P.T. Vanathi, A review of vector quantization techniques, IEEE Potentials, vol. 25, 2006, pp. 39-47. - [32] R. Gray, Vector quantization, IEEE Assp Magazine, vol. 1, 1984, pp. 4-29. - [33] A. Razavi, A. van den Oord, and O. Vinyals, Generating diverse highfidelity images with VQ-VAE-2, 2019, arXiv preprint arXiv:1906.00446. - [34] P. Dhariwal, H. Jun, C. Payne, J. W. Kim, A. Radford, and I. Sutskever, Jukebox: A generative model for music, 2020, arXiv preprint arXiv:2005.00341. - [35] J. MacQueen, Some methods for classification and analysis of multivariate observations, Proc. 5th Berkeley Symp. Math. Statist. Probability. (1967) 281-297. - [36] S. Welker, M. Le, R. T. Q. Chen, W. Hsu, T. Gerkmann, A. Richard, and Y. Wu, FlowDec: A flow-based full-band general audio codec with high perceptual quality, 2025, arXiv preprint arXiv:2503.01485. - [37] J. Yao, H. Liu, C. Chen, Y. Hu, ES Chng, L Xie, GenSE: Generative Speech Enhancement via Language Models using Hierarchical Modeling, 2025, arXiv preprint arXiv:2502.02942. - [38] W. Jang, D. Lim, J. Yoon, B. Kim, and J. Kim, Univnet: A neural vocoder with multi-resolution spectrogram discriminators for high-fidelity waveform generation, in: Proc. Interspeech, 2021, pp. 2207-2211. - [39] Y. Bengio, N. Léonard, and A. Courville, Estimating or propagating gradients through stochastic neurons for conditional computation, 2013, arXiv preprint arXiv:1308.3432. - [40] N. Shazeer, Y. Cheng, N. Parmar, D. Tran, A. Vaswani, P. Koanantakool, P. Hawkins, H. Lee, M. Hong, C. Young, et al., Mesh-tensorflow: Deep learning for supercomputers, Advances in Neural Information Processing Systems. 31 (2018) 10414-10423. - [41] L. Wang, H. Gao, C. Zhao, X. Sun, and D. Dai , Auxiliary-loss-free load balancing strategy for mixture-of-experts, 2024, arXiv preprint arXiv:2408.15664. - [42] J. Kong, J. Kim, and J. Bae, Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis, Advances in neural information processing systems. 33 (2020) 17022-17033. - [43] I. Loshchilov and F. Hutter, Decoupled weight decay regularization, in: International Conference on Learning Representations. (2019). - [44] M. Schoeffler, F. Stöter, B. Edler, and J. Herre, Towards the next generation of web-based experiments: A case study assessing basic audio quality following the ITU-R recommendation BS. 1534 (MUSHRA), in: 1st Web Audio Conference, 2015, pp. 1-6. - [45] A. Rix, J. Beerends, M. Hollier, and A. Hekstra, Perceptual evaluation of speech quality (PESQ)-a new method for speech quality assessment of telephone networks and codecs, in: 2001 IEEE international conference on acoustics, speech, and signal processing(ICASSP), IEEE, 2001, pp. 749-752. - [46] M. Chinen, F. Lim, J. Skoglund, N. Gureev, F. O’Gorman, and A. Hines., Visqol v3: An open source production ready objective speech and audio metric, in: 2020 twelfth international conference on quality of multimedia experience (QoMEX), IEEE, 2020, pp. 1-6. - [47] C. Veaux, J. Yamagishi, K. MacDonald et al., Cstr vctk corpus: English multi-speaker corpus for cstr voice cloning toolkit, University of Edinburgh. The Centre for Speech Technology Research (CSTR), vol. 6, 2017, p. 15. - [48] H. Zen, V. Dang, R. Clark, Y. Zhang, R. Weiss, Y. Jia, Z. Chen, and Y. Wu, LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech, in: Proc. Interspeech, 2019, pp. 1526-1530. - [49] R. Ardila, M. Branson, K. Davis, M. Henretty, M. Kohler, J. Meyer, R. Morais, L. Saunders, F. Tyers, and G. Weber, Common voice: A massively-multilingual speech corpus, 2019, arXiv preprint arXiv:1912.06670. - [50] M. Defferrard, K. Benzi, P. Vandergheynst, and X. Bresson, FMA: A Dataset For Music Analysis, in: 18th International Society for Music Information Retrieval Conference, 2017.