2410.02056v1

# Unknown Title ## SYNTHIO: AUGMENTING SMALL-SCALE AUDIO CLASSIFICATION DATASETS WITH SYNTHETIC DATA Sreyan Ghosh ♦♠ Sonal Kumar ♠ Zhifeng Kong ♦ Rafael Valle ♦ Bryan Catanzaro ♦ Dinesh Manocha ♠ ♦ NVIDIA, CA, USA ♠ University of Maryland, College Park, USA ## ABSTRACT We present Synthio , a novel approach for augmenting small-scale audio 1 classification datasets with synthetic data. Our goal is to improve audio classification accuracy with limited labeled data. Traditional data augmentation techniques, which apply artificial transformations (e.g., adding random noise or masking segments), struggle to create data that captures the true diversity present in real-world audios. To address this shortcoming, we propose to augment the dataset with synthetic audio generated from text-to-audio (T2A) diffusion models. However, synthesizing effective augmentations is challenging because not only should the generated data be acoustically consistent with the underlying small-scale dataset, but they should also have sufficient compositional diversity . To overcome the first challenge, we align the generations of the T2A model with the small-scale dataset using preference optimization. This ensures that the acoustic characteristics of the generated data remain consistent with the small-scale dataset. To address the second challenge, we propose a novel caption generation technique that leverages the reasoning capabilities of Large Language Models to (1) generate diverse and meaningful audio captions and (2) iteratively refine their quality. The generated captions are then used to prompt the aligned T2A model. We extensively evaluate Synthio on ten datasets and four simulated limited-data settings. Results indicate our method consistently outperforms all baselines by 0.1%-39% using a T2A model trained only on weakly-captioned AudioSet. Code will be available here. ## 1 INTRODUCTION Audio classification is the foundational audio processing task of understanding the input audio and assigning it to one or multiple predefined labels. However, training audio classification models requires a lot of high-quality labeled data, which is not always readily available (Ghosh et al., 2022). Manually collecting and annotating large-scale audio datasets is an expensive, time-consuming, and noisy process (Nguyen et al., 2017; Mart´ ın-Morat´ o & Mesaros, 2021), and recent concerns about data privacy and usage rights further hinder this process (Ren et al., 2023). Data augmentation, which involves expanding original small-scale datasets with additional data, is a promising solution to address data scarcity. Traditional augmentation techniques attempt to diversify audio samples by applying randomly parameterized artificial transformations to existing audio. These methods include spectral masking (Park et al., 2019), temporal jittering (Nanni et al., 2020), cropping (Niizumi et al., 2021), mixing (Seth et al., 2023; Ghosh et al., 2023b; Niizumi et al., 2021) and other techniques (Saeed et al., 2021; Al-Tahan & Mohsenzadeh, 2021; Manocha et al., 2021). While these approaches have shown success, they operate at the level of observed data rather than reflecting the underlying data-generating process that occurs in real-world scenarios. As a result, they statistically modify the data without directly influencing the causal mechanisms that produced it, leading to high correlations between augmented samples and limited control over diversity. Generating synthetic data from pre-trained text-to-audio (T2A) models addresses the limitations of standard data augmentation techniques while retaining their strengths of universality , controllability , and performance (Trabucco et al., 2024). The recent success of generative models makes this approach particularly appealing (Long et al., 2024; Evans et al., 2024b). However, generating synthetic audio presents unique challenges due to the complexity of waveforms and temporal 1 We use 'audio' to refer to acoustic events comprising non-verbal speech, non-speech sounds, and music. dependencies (Ghosh et al., 2024b). We highlight the 3 main challenges in generating effective synthetic data for audio classification: i) Consistency with the original data: Synthetic audio that does not align acoustically with the original dataset can hinder effective augmentation and may cause catastrophic forgetting (Geiping et al., 2022). This misalignment includes spectral, harmonic, and other inherent acoustic characteristics not easily controlled through prompts. Maintaining consistency with T2A models trained on internet-scale data remains a challenge, and standard fine-tuning can often lead to overfitting (Weili et al., 2024). ii) Diversity of generated data: Ensuring compositional diversity in the generated synthetic data (e.g., sound events, temporal relationships, background elements, etc.) is critical for effective augmentation. Additionally, a lack of diversity can lead to poor generalization and learning of spurious correlations, impacting performance. Simple, hand-crafted prompts (e.g., 'Sound of a metro') often result in repetitive patterns, and creating diverse, meaningful prompts is labor-intensive. Complex prompts can generate audios that do not preserve the original label. iii) Limitations of current T2A models: T2A models often struggle to generate diverse audios and follow details in prompts. This is largely due to the lack of large-scale, open-source datasets for training, as well as the inherent complexity of non-speech audio domains (Ghosal et al., 2023). These limitations highlight the need for more advanced approaches for synthetic data generation in audio. Our Contributions. To address these challenges, we propose Synthio , a novel, controllable and scalable approach for augmenting small-scale audio classification datasets with synthetic data. Our proposed approach has 2 main steps: i) Aligning the Text-to-Audio Models with Preference Optimization: To generate synthetic audios with acoustic characteristics consistent with the small-scale dataset, we introduce the concept of aligning teaching with learning preferences . Specifically, we align the generations of the T2A model (acting as the teacher) with the target characteristics of the small-scale dataset using preference optimization. This approach ensures that the synthetic audios reflect the acoustic properties of (or sound similar to ) the downstream dataset, enabling the classification model (the student) to perform well on test data with similar characteristics. To achieve this, we train a diffusion-based T2A model with preference optimization, where audios generated from Gaussian noise are treated as losers and audios from the downstream dataset are treated as winners. Figure 1: Performance comparison of Synthio with other augmentation methods on down-sampled ESC50 (100 samples). Traditional augmentation, such as SpecAug, degrades performance on small-scale datasets. Naive synthetic augmentation outperforms traditional methods significantly but plateaus with higher sample counts. Synthio further enhances performance by generating consistent and diverse synthetic data. <details> <summary>Image 1 Details</summary>  ### Visual Description ## Line Graph: Classification Accuracy vs. Number of Generated Augmentations ### Overview The graph compares classification accuracy across four augmentation strategies as the number of generated augmentations increases from 0 to 500. Four data series are plotted with distinct markers and line styles, showing performance trends over augmentation volume. ### Components/Axes - **X-axis**: "Number of Generated Augmentations" (0 to 500, linear scale) - **Y-axis**: "Classification Accuracy" (0.4 to 0.9, linear scale) - **Legend**: Located in the top-left corner, with four entries: - **No Augmentation**: Green circle (dashed line) - **SpecAug**: Red diamond (dotted line) - **Vanilla Syn. Aug.**: Blue triangle (dashed line) - **Synthio (ours)**: Red cross (solid line) ### Detailed Analysis 1. **No Augmentation** (Green): - Starts at ~0.55 at 0 augmentations. - Remains flat throughout, ending at ~0.55 at 500 augmentations. - Uncertainty: ±0.01 (flatline suggests minimal variance). 2. **SpecAug** (Red Diamonds): - Begins at ~0.48 at 0 augmentations. - Sharp increase to ~0.78 at 100 augmentations. - Plateaus at ~0.78 for all subsequent points (200–500). - Uncertainty: ±0.01 (stable after 100). 3. **Vanilla Syn. Aug.** (Blue Triangles): - Starts at ~0.55 at 0 augmentations. - Rises to ~0.78 at 200 augmentations. - Slight decline to ~0.72 at 500 augmentations. - Uncertainty: ±0.01 (consistent trend). 4. **Synthio (Ours)** (Red Crosses): - Begins at ~0.55 at 0 augmentations. - Sharp rise to ~0.82 at 100 augmentations. - Maintains ~0.82 for all subsequent points (200–500). - Uncertainty: ±0.01 (stable after 100). ### Key Observations - **Synthio (Ours)** achieves the highest accuracy (~0.82) after 100 augmentations, outperforming all other methods. - **SpecAug** shows a significant early improvement (~0.48 → 0.78) but plateaus below Synthio. - **Vanilla Syn. Aug.** peaks at ~0.78 but declines slightly (~0.72) at 500 augmentations, suggesting potential overfitting. - **No Augmentation** remains the baseline (~0.55) with no improvement. ### Interpretation The data demonstrates that augmentation strategies significantly improve classification accuracy compared to no augmentation. **Synthio (Ours)** is the most effective method, achieving a 47% relative improvement over the baseline (0.55 → 0.82). **SpecAug** and **Vanilla Syn. Aug.** also enhance performance but with diminishing returns or instability at higher augmentation counts. The decline in Vanilla Syn. Aug. at 500 augmentations may indicate overfitting or saturation effects. These trends suggest that Synthio’s augmentation approach optimizes data diversity without compromising generalization, making it superior for this task. </details> ii) Generating Diverse Synthetic Augmentations: To generate diverse audios for augmentation, we introduce the concept of language-guided audio imagination and imagine novel acoustic scenes with language guidance. Specifically, we generate diverse audio captions that are then used to prompt T2A models to generate audios with varied compositions. To achieve this, we propose MixCap , where we prompt LLMs iteratively to generate captions combining existing and new acoustic components. Additionally, we employ a self-reflection module that filters generated captions and prompts the LLM to revise those that do not align with the intended label. To summarize, our main contributions are: 1. We introduce Synthio , a novel data augmentation approach for audio classification that expands small-scale datasets with synthetic data. Synthio uses novel methods to tackle the inherent challenges of producing consistent and diverse synthetic data from T2A models. 2. We evaluate Synthio across 10 datasets in 4 simulated low-resource settings, demonstrating that, even with a T2A model trained on weakly captioned AudioSet, Synthio outperforms all baselines by 0.1%-39%. 3. We conduct an in-depth analysis of the generated augmentations, highlighting Synthio's ability to produce diverse and consistent data, its scalability, and its strong performance on complex tasks such as audio captioning. ## 2 RELATED WORK Data Augmentation for Audio and Beyond. Expanding or augmenting small-scale datasets with additional data has been widely studied in the literature. Traditional augmentation methods, which apply randomly parameterized artificial transformations to data during training, remain the most common approach across language Wei & Zou (2019); Karimi et al. (2021), vision (Shorten & Khoshgoftaar, 2019; Wang et al., 2017; Yun et al., 2019), and audio (Park et al., 2019; Spijkervet, 2021). For audio, specific techniques include SpecAugment, adding background noise, reverberation, and random spectrogram transformations. With the emergence of generative models, synthetic data augmentation has been increasingly adopted for language (Ghosh et al., 2023a; 2024c; Chen et al., 2021) and vision (Trabucco et al., 2024; Zhao et al., 2024; He et al., 2023), proving to be more effective than traditional methods. These approaches generally incorporate explicit steps to ensure the consistency and diversity of generated augmentations. While some progress has been made in speech, synthetic data for audio remains relatively underexplored. Recently, parallel work has investigated simple prompting of T2A models to improve T2A generation (Kong et al., 2024) and environmental scene classification (Ronchini et al., 2024). Text-to-Audio Generation. In recent years, there has been a significant surge in research on textto-audio (T2A) models. The most popular architectures include auto-regressive models based on codecs (Kreuk et al., 2023; Copet et al., 2024) and diffusion models Liu et al. (2023); Ghosal et al. (2023); Evans et al. (2024a). Clotho (Drossos et al., 2020) and AudioCaps (Kim et al., 2019) remain the largest human-annotated datasets for training these models. However, large-scale datasets for T2A model training are still scarce. Recently, Yuan et al. (2024) synthetically captioned AudioSet (Gemmeke et al., 2017), demonstrating its effectiveness for training T2A models. For downstream adaptation, earlier works have primarily relied on Empirical Risk Minimization (ERM). Majumder et al. (2024) introduced preference optimization for T2A models, creating a synthetic preference dataset based on scores provided by a CLAP model (Elizalde et al., 2023). ## 3 BACKGROUND Diffusion Models. Diffusion models consist of two main processes: a forward process and a reverse process. Given a data point x 0 with probability distribution p ( x 0 ) , the forward diffusion process gradually adds Gaussian noise to x 0 according to a pre-set variance schedule β 1 , ⋯ , β T and degrades the structure of the data. At the time step t , the latent variable x t is only determined by the x t -1 due to its discrete-time Markov process nature. At inference time, the diffusion model iteratively executes the reverse process T times starting from a randomly sampled Gaussian Noise ( ϵ ∼ N ( 0 , I )) . For more details on diffusion models, we request our readers to refer to Appendix A.1. We now explain Direct Preference Optimization (DPO), a simpler alternative to RLHF (Christiano et al., 2017). Abusing notation, we will also x 0 as random variables for language. Reward Modeling. Estimating human preferences for a particular generation x 0 , given the context c , is challenging because we do not have direct access to a reward model r ( c, x 0 ) . In our scenario, we assume only ranked pairs of samples are available, where one sample is considered a 'winner' ( x w 0 ) and the other a 'loser' ( x l 0 ) under the same conditioning c . Based on the Bradley-Terry (BT) model, human preferences can be modeled as: $$P B ( x _ { 0 } > x _ { 0 } | c ) = o ( r ( c , x _ { 0 } ) - r$$ where σ represents the sigmoid function. The reward model r ( c, x 0 ) is parameterized by a neural network ϕ and trained through maximum likelihood estimation for binary classification: $$L _ { B r } ( \phi ) = - E _ { c , x , y } ^ { 2 } + [ | \log \sigma ( r _ { 0 } ) | ( c _ { 0 } ^ { 2 } ) ]$$ Here, prompt c and data pairs ( x w 0 , x l 0 ) are drawn from a dataset labeled with human preferences. RLHF : The goal of RLHF is to optimize a conditional distribution p θ ( x 0 ∣ c ) , where c ∼ D c , such that the latent reward model r ( c, x 0 ) is maximized. This is done while regularizing the distribution through the Kullback-Leibler (KL) divergence from a reference distribution p ref , resulting in the following objective: $$\max E _ { p 0 } \left[ - x _ { 0 } - r _ { 0 } \rho ( z _ { 0 } ) \right] - \beta D _ { p 0 }$$ Here, the hyperparameter β controls the strength of regularization. DPO : DPO directly optimizes the conditional distribution p θ ( x 0 ∣ c ) to align data generation with the preferences observed in (any form of) feedback. The goal is to optimize the distribution of generated data such that it maximizes alignment with human preference rankings while maintaining consistency with the underlying reference distribution p ref ( x 0 ∣ c ) . The optimal solution p ∗ θ ( x 0 ∣ c ) for the DPO objective can be expressed as: $$\rho ^ { \ast } ( x _ { 0 } | c ) = p _ { ref } ( x _ { 0 } | c ) - \frac { e x p ( r | c , z _ { 0 } ) } { Z ( c ) }$$ where Z ( c ) is the partition function, defined as: $$z ( c ) = \sum _ { x _ { 0 } } ^ { n } p _ { ref } ( x _ { 0 } | e ) e x p ( r ( c , x _ { 0 } )$$ This term ensures proper normalization of the distribution, and β controls the regularization, balancing between adherence to the reference distribution and preference maximization. The reward function r ( c, x 0 ) is then reparameterized as: $$r ( c , x _ { 0 } ) = \beta log \frac { p ^ { a } ( x _ { 0 } | c ) } { p _ { ref } ( x _ { 0 } | c ) } + \beta l _ { 0 }$$ Using this reparameterization, the reward objective can be formulated as: $$\frac { L_{dro}( \theta ) } { 1 - e^{-\beta log \sigma ( \beta log \rho _{e} ( \theta ) ) } } = - E _ { c , n } ^ { \alpha } p _ { 0 } ( x _ { 0 } | c ) \log \sigma ( \beta log \rho _ { e } ( \theta ) )$$ By optimizing this objective, DPO enables direct preference learning, optimizing the conditional distribution p θ ( x 0 ∣ c ) in such a way that it better reflects human preferences, as opposed to traditional approaches that optimize the reward function first and then perform reinforcement learning. DPO for Diffusion Models: Very recently, Wallace et al. propose a formulation for optimizing diffusion models with DPO. The primary issue with optimizing diffusion with DPO is that the distribution p θ ( x 0 ∣ c ) is not tractable due to the need to consider all possible diffusion paths leading to x 0 . To address this, Wallace et al. propose to leverage the evidence lower bound (ELBO) to incorporate latents x 1 ∶ T , which represent the diffusion path. The reward R ( c, x 0 ∶ T ) accounts for the entire sequence, leading to the reward function: $$r ( c , x _ { 0 } ) = E _ { p _ { 0 } } ( x _ { 1 } ; T _ { z _ { 0 } } o _ { 1 } [ R ( c , x _ { 0 } ) ]$$ Instead of directly minimizing the KL-divergence as typically done, they propose to utlize the upper bound of the joint KL-divergence D KL [ p θ ( x 0 ∶ T ∣ c )∣∣ p ref ( x 0 ∶ T ∣ c )] . This is integrated into the optimization objective, enhancing the practicality of training diffusion models with preferences. The new objective, aiming to maximize the reward and match the distribution of the reverse process of p θ to the reference model p ref , is given by: $$\max _ { p _ { 0 } } E _ { c , x _ { 0 } } - p _ { 0 } ( x _ { 0 } \cdot T [ c ] ) | r$$ Training efficiency is improved by approximating the intractable reverse process using a forward approximation q ( x 1 ∶ T ∣ x 0 ) . The DPO then integrates this into the loss function, which involves comparing the log likelihood ratio of the probabilities under p θ and p ref for winning and losing paths: $$\frac { L _ { DPO-Diffusion } ( \theta ) = - E _ { c , x _ { 0 } } } { p _ { ref } ( x _ { 1 } : T [ x _ { 0 } ] ) }$$ After applying Jensen's inequality to take advantage of the convexity of -log σ , we push the expectation outside, allowing us to simplify the objective. By approximating the denoising process with the forward process, the final form of the loss for DPO in diffusion models, in terms of the L2 noise estimation losses, becomes: $$\log \sigma ( - \beta T_c ( \lambda _ { 1 } ) \Delta l )$$ where ∆ L is the L2 weighted noise estimation losses between the preferred (winner) and less preferred (loser) samples. ## 4 METHODOLOGY Let D small = {( a i , l i ) , 1 ≤ i ≤ n } be a high-quality, small-scale human-annotated audio classification dataset with n audio-label pairs. Let D a-c be a potentially noisy, large-scale weakly-captioned dataset of audio-caption pairs with zero intersection with D small. Our goal is to train a T2A model T θ using D a-c , then use it to generate a synthetic dataset D syn and then finally add it to D small (now attributed as D train) to improve audio classification performance. This is accomplished through two key steps: first, aligning the generations from T θ with the acoustic characteristics of D small, and second, generating new captions to prompt T θ for creating synthetic audio data. We elaborate on these next. ## 4.1 ALIGNING THE TEXT-TO-AUDIO MODEL USING PREFERENCE OPTIMIZATION T2A models trained on internet-scale data often generate audio that diverges from the characteristics of small-scale datasets, resulting in distribution shifts. These mismatches can include variations in spectral (e.g., frequency content), perceptual (e.g., pitch, loudness), harmonic, or other acoustic characteristics 2 . This misalignment arises from the non-deterministic nature of T2A generation and it is impractical to provide detailed attributes (like 'loud' or 'high-pitched') in prompts, as (i) there are no scalable methods for extracting specific attributes for each label, and (ii) T2A models struggle with accurately following finegrained prompt details (Wang et al., 2024). To address these issues, we propose the concept of aligning teaching with learning preferences . Our approach assumes that the classification model (viewed as the student) performs better when trained on synthetic audio that closely matches the inherent acoustic Figure 2: We propose to align the T2A model T θ with the small-scale dataset D small using DPO. This helps us generate audios with acoustic characteristics aligned to that of D small . <details> <summary>Image 2 Details</summary>  ### Visual Description ## Diagram: Text-to-Audio Model Training Pipeline ### Overview This diagram illustrates a technical pipeline for training a text-to-audio model using preference-based optimization. It shows interactions between models, databases, and training processes to improve audio generation quality. ### Components/Axes 1. **Models**: - **Text-to-Audio Model** (central component, generates audio from text). - **Aligned Text-to-Audio Model** (updated via preference optimization). 2. **Databases**: - **D_small**: Source of text conditions and random Gaussian noise. - **D_pref**: Stores preference data (winning audios). - **Generated Audios**: Output from the Text-to-Audio Model. - **Losing Audios**: Subset of generated audios not aligned with preferences. 3. **Processes**: - **Preference Optimization**: Updates the Aligned Model using winning audios. - **Adapters Training**: Trains adapters using losing audios. 4. **Arrows/Relationships**: - Dashed red arrow: Preference optimization direction (↑). - Solid blue arrows: Flow of winning/losing audios. - Dotted line: Adapters training process. ### Detailed Analysis - **Text Condition**: Input to the Text-to-Audio Model (e.g., "Sound of a [label]"). - **Random Gaussian Noise**: Added to text conditions to diversify audio generation. - **Winning Audios**: Generated audios that align with preferences (stored in D_pref). - **Losing Audios**: Generated audios that fail to meet preferences (used for adapters training). - **τθ (theta)**: Parameter updated during preference optimization (↑). ### Key Observations 1. The pipeline iteratively improves the Aligned Model by prioritizing audios that match user preferences (D_pref). 2. Losing audios are repurposed for adapters training, suggesting a dual-path optimization strategy. 3. Random Gaussian noise introduces variability in text-to-audio generation, potentially enhancing robustness. ### Interpretation This system demonstrates a **reinforcement learning**-like approach where the model self-corrects by: - Using **preference data** (D_pref) to refine audio alignment with user expectations. - Leveraging **losing audios** to train adapters, addressing edge cases or underrepresented scenarios. - Balancing **exploration** (via noise injection) and **exploitation** (via preference optimization). The diagram emphasizes a closed-loop system where generated audios are continuously evaluated and refined, ensuring the model evolves toward higher-quality outputs. The absence of explicit numerical values suggests a conceptual framework rather than empirical data. </details> properties of our high-quality and human-labeled D small. Thus, we align the generations of the T2A model (viewed as the teacher) to D small, ensuring that the generated augmentations align with the desired characteristics and sound similar , ultimately enhancing the student model's ability to generalize to similarly characterized test data. As shown in Fig. 2, we achieve this using preference optimization (DPO in our case) and align generations of T θ with D small. Unlike standard fine-tuning, which can lead to less diverse outputs and overfitting due to a narrow focus on minimizing loss, preference optimization encourages greater exploration in the model's output space, preventing mode collapse and fostering more diverse augmentations. Additionally, DPO leverages pairwise learning, offering richer training signals compared to the independent outputs used in standard fine-tuning, further mitigating overfitting risks. We detail our two-step approach for DPO optimization below: Step 1: Construction of the Preference Dataset. To create our preference dataset D pref = {( a w 1 , a l 1 ) , ⋯ , ( a w j , a l j )} , we first generate template-based captions for each instance in D small in the form: 'Sound of a label ', where label is the category associated with the audio. For each instance, we prompt the T2A model j times, with all generations starting from randomly initialized Gaussian noise (generation configuration is detailed in Section 5). Each generated audio is then paired with the corresponding ground-truth audio from the gold dataset. This resulting D pref dataset has n × j instances, where the generated audio is treated as the 'loser' and the ground-truth audio as the 'winner'. This simple approach has proven highly effective in aligning generations by generative models by prior work (Majumder et al., 2024; Tian et al., 2024). Step 2: Preference Optimization Using DPO. After constructing D pref, we train our T2A model on this dataset with DPO using the approach outlined in Section 3. The resulting aligned model is referred to as T θ aln . Details of the hyper-parameters used for training are provided in Section 5. ## 4.2 GENERATING DIVERSE SYNTHETIC AUGMENTATIONS It is not well-studied in the literature on how to leverage synthetic audio generation for downstream tasks. The only existing work relied on manually crafted prompt templates (e.g., 'Sound of a { label } ') (Ronchini et al., 2024). It has a significant limitation: there is no precise control over the specific components in the generated audio for a given caption. This can result in repetitive or completely inconsistent patterns, particularly with weaker T2A models 3 . These could bias the model to learn spurious correlations, a known issue in synthetic data augmentation (Ghosh et al., 2024c). 2 When prompted with 'sound of a bus ' for the category ' bus ' in the TUT-Urban dataset, the generated audio may not reflect the typical bus sounds in European cities (where TUT was recorded), as bus sounds can vary by region, with some featuring loud engines and dense crowds while others have quieter engines and sparse crowds. 3 For example, when prompted with 'Sound of a park ', we observed that 9 out of 10 times, the model generated the sound of children playing as part of the generated audio. On the other hand, when prompted with 'Sound of a airport ', the model generates audios with background announcements, which could vary by regions. Figure 3: Overview of our proposed Language-Guided Audio Imagination for generating diverse synthetic augmentations. Starting with the small-scale dataset, we first generate audio captions and use an LLM to extract acoustic components (Prompt 1). Using these components and audio labels, we prompt the LLM to generate new and diverse captions (Prompt 2), which are then used to prompt the aligned T2A model for audio generation. The generated audios are filtered for label consistency using CLAP, with accepted audios added to the final synthetic dataset. Rejected audios undergo caption revision (Prompt 3) through a self-reflection process, and the revised captions are used to regenerate audios, iterating this process i times. Example captions are in Table 6. <details> <summary>Image 3 Details</summary>  ### Visual Description ## Flowchart: Audio Captioning and LLM Integration System ### Overview The diagram illustrates a cyclical system for processing audio captions using Large Language Models (LLMs), acoustic components, and self-reflection mechanisms. It emphasizes iterative refinement of audio data through filtering, feedback loops, and text-to-audio generation. ### Components/Axes 1. **Key Components**: - **Audio Captioning Model**: Processes audio captions using a small-scale dataset and synthetic data. - **LLM (Large Language Model)**: Interacts with prompts (Prompt 1, Prompt 2, Prompt 3) and existing acoustic components. - **MixCap**: A module connecting LLMs to acoustic components. - **CLAP Filtering**: Filters accepted/rejected audios. - **Self-Reflection**: A feedback loop for iterative improvement. - **Text-to-Audio Model**: Generates new audios from new captions. 2. **Data Flow**: - **Inputs**: Audio Captions, Small-Scale Dataset, Synthetic Data. - **Outputs**: Accepted Audios, Rejected Audios, New Audio Captions, Generated Audios. 3. **States**: - **Trainable/Frozen**: Indicates adjustable vs. static components (e.g., LLM, CLAP Filtering). ### Detailed Analysis - **Audio Captioning Model**: - Takes audio captions as input. - Outputs accepted/rejected audios after processing small-scale and synthetic datasets. - Includes an "AST" (possibly an attention mechanism or algorithm) marked as "Trainable." - **LLM and MixCap**: - LLM processes prompts (Prompt 1, Prompt 2, Prompt 3) and existing acoustic components. - MixCap acts as a bridge between LLM and acoustic components, enabling integration. - **CLAP Filtering**: - Accepts/rejects audios based on criteria (e.g., relevance, quality). - Rejected audios re-enter the system via Self-Reflection. - **Self-Reflection**: - A feedback loop where rejected audios and generated audios are reprocessed. - Central to iterative refinement of outputs. - **Text-to-Audio Model**: - Converts new audio captions into generated audios. - Outputs feed back into the system for further processing. ### Key Observations 1. **Cyclical Workflow**: The system emphasizes continuous improvement through feedback loops (e.g., Self-Reflection). 2. **Hybrid Data Use**: Combines small-scale datasets with synthetic data for robustness. 3. **LLM-Centric Design**: LLMs drive prompt-based processing and integration with acoustic components. 4. **Filtering Mechanisms**: CLAP Filtering ensures quality control by rejecting suboptimal audios. 5. **Dynamic Adaptation**: The "Trainable" state of components suggests adaptability to new data. ### Interpretation This system demonstrates a sophisticated approach to audio captioning and generation, leveraging LLMs for contextual understanding and acoustic components for technical processing. The integration of CLAP Filtering and Self-Reflection highlights a focus on quality assurance and iterative learning. The use of synthetic data alongside small-scale datasets suggests scalability and diversity in training. The Text-to-Audio Model’s role in generating new audios indicates a closed-loop system capable of evolving with new inputs. The "Trainable" state of critical components (e.g., AST, LLM) implies ongoing optimization, making the system adaptable to changing requirements or data distributions. </details> While the alignment stage helps the T2A model generate audio with acoustic characteristics similar to the small-scale dataset (e.g., spectral, harmonic, etc.), it does not fully account for the compositional diversity of the generated audios (e.g., sound events, their temporal relationships, background elements). To tackle this, we propose the concept of language-guided audio imagination , where we propose to imagine novel audios guided by language. Specifically, we leverage the reasoning abilities of LLMs to generate diverse and meaningful captions for a category label in a controlled yet scalable manner. These captions are then used to prompt our aligned T2A model for generating novel audios. ## 4.2.1 GENERATING DIVERSE PROMPTS WITH MIXCAP We propose MixCap , a prompt generation method that creates diverse and effective captions in three steps: First, we employ GAMA (Ghosh et al., 2024a) to caption all audio files in D small . Next, we prompt an LLM to extract phrases describing the acoustic components of the audio. These components correspond to the acoustic elements such as backgrounds and foreground events, and their attributes and relations, etc (see prompt in Appendix A.2). Finally, for each training instance in D small , we prompt the LLM with the ground-truth label and the extracted components from all instances to generate N diverse audio captions that blend existing and new components. This approach prevents overemphasis on events already present in D small and encourages diversity in the generated audio. These captions are then used to prompt the aligned T2A model to generate synthetic data D syn . ## 4.2.2 FILTERING & SELF-REFLECTION Filtering. After generating captions and their corresponding audio, we filter the audio for label consistency. While LLMs can generate diverse captions, the audio produced must remain aligned with the ground-truth label. To ensure this, we use CLAP to evaluate the generated audio, accepting those that meet a similarity threshold of p % and rejecting the rest. We denote the accepted audios as D acc syn and the rejected ones as D rej syn . The CLAP model is pre-trained on D a-c and fine-tuned on D small to adapt to the target dataset. Example captions are in Table 6, and prompts are in Appendix A.2. Self-Reflection. For the rejected audios in D rej syn , we prompt the LLM to reflect on its generated captions and revise them to better align with the target label. Precisely, we feed the LLM with the original caption of each rejected audio along with extracted components from all accepted captions in D acc syn and task it to rewrite the rejected captions. The revised captions are then used to generate new audio, which is again filtered using CLAP. Audios that meet the threshold are added to D acc syn ; those that do not are returned to D rej syn . This process repeats for i iterations or until D rej syn is empty. We illustrate examples of generated and revised captions in Table 6. Fine-tuning for Audio Classification. After the self-reflection stage, the final set of accepted synthetic audios is denoted as D syn, containing ≈ N × n audio-label pairs, where N represents the augmentation factor (e.g., with 100 gold samples, we generate 100 × N synthetic samples). This set is then combined with D small to form the final training dataset D train , which is then used to train the audio classification model. ## 5 EXPERIMENTAL SETUP Models and Hyper-Parameters. For our T2A model, we choose the Stable Audio architecture (Evans et al., 2024b). Stable Audio is based on the Diffusion Transformer architecture, and we refer our readers to the original paper for more details. We train the model from scratch on SoundVECaps (Yuan et al., 2024) (with ≈ 1.5 million weakly captioned audio-caption pairs) to avoid any data leakage. For training, we employ a batch size of 64, an AdamW optimizer, a learning rate of 5e-4, and a weight decay of 1e-3 for 40 epochs. For DPO-based alignment tuning, we fine-tune with a batch size of 32 and a learning rate of 5e-4 for 12 epochs. For our audio classification model, we employ the Audio Spectrogram Transformer (AST) (Gong et al., 2021) (pre-trained on the AudioSet dataset) and fine-tune it with a batch size of 24 and learning rate of 1e-4 for 50 epochs. In each experiment, we adjust the number of generated augmentations N (ranging from 1 to 5) based on performance on the validation set. All hyper-parameters are fixed across splits, and results are averaged across 3 runs. Datasets. We create small-scale datasets by downsampling commonly used audio classification datasets to n samples. Our selected datasets include a mix of music, everyday sounds, and acoustic scenes. For multi-class classification, we use NSynth Instruments, TUT Urban, ESC50 (Piczak), USD8K (Salamon et al., 2014), GTZAN (Tzanetakis et al., 2001), Medley-solos-DB (Lostanlen & Cella, 2017), MUSDB18 (Rafii et al., 2017), DCASE Task 4 (Mesaros et al., 2017), and Vocal Sounds (VS) (Mesaros et al., 2017), evaluating them for accuracy. For multi-label classification, we use the FSD50K (Fonseca et al., 2022) dataset and evaluate it using the F macro 1 metric. We exclude AudioSet from evaluation as Sound-VECaps is derived from it. To ensure a downsampled dataset that has a label distribution similar to that of the of the original dataset, we employ stratified sampling based on categories. Our experiments are conducted with n = { 50, 100, 200, 500 } samples, and we downsample the validation sets for training while evaluating all models on the original test splits. Baselines. Our baselines include: (i) Gold-only (No Aug.): We employ only the small-scale dataset for training and do not perform any augmentations. (ii) Traditional augmentation baselines: SpecAugment, Noise Augmentation (we either add random Gaussian noise or background noise from AudioSet and present averaged results), Pitch and Time Shift and Audiomentations (Jordal, 2021) - a combination of the AddGaussianNoise, TimeStretch, PitchShift, Shift, SpecFrequencyMask, TimeMask and TimeStretch - combination with the highest average score on 4 datasets and splits and was selected after grid search over all possible combinations). (iii) Generative baselines: Vanilla Synthetic Augmentation (Vanilla Syn. Aug.) - we prompt T θ with template captions), Vanilla Syn. Aug. + LLM Caps - we prompt T θ with random captions generated with LLMs. (iv) Finally, inspired by Burg et al. (2023), we also employ a retrieval baseline where instead of generating augmentations from our T2A model trained on D a-c , we just retrieve the topn instances (w.r.t. CLAP similarity) from the AudioSet for each instance in D small as our augmentations. Ablations. Weablate Synthio with: (i) w/o Self-Reflection: We remove the repetitive self-reflection module and iterate and filter only once; (ii) w/o DPO: We skip the tuning step and prompt the un-alined T θ for augmentations; (iii) w/ ERM: We replace DPO tuning with standard Empirical Risk Minimization(ERM)-based fine-tuning with diffusion loss; (iv) w/ Template Captions: We remove MixCap and self-reflection modules and prompt T θ aln with template captions; (v) w/o MixCap: Similar to our Random Captions baseline, but we retain all other modules of Synthio. ## 6 RESULTS AND DISCUSSION In this section, we present our main benchmark results, followed by detailed analyses to highlight Synthio's strengths and explore its versatility. Key areas of investigation include Synthio's consistency and diversity in augmentations, performance on synthetic-only datasets, scalability, impact on longtailed categories, and its potential extension to complex audio understanding tasks like captioning. Table 1: Result comparison of Synthio with baselines on 10 datasets and 4 small-scale settings. n refers to the number of samples in the small-scale dataset augmented with synthetic data. Synthio outperforms our baselines by 0.1% - 39%. We also highlight the relative improvements by Synthio compared to the Gold-only. | n | Method | ESC-50 | USD8K | GTZAN | Medley | TUT | NSynth | VS | MSDB | DCASE | FSD50K | |-----|-------------------------------------|------------------|------------------|------------------|------------------|------------------|------------------|-------------------|------------------|------------------|------------------| | | Gold-only (No Aug.) | 22.25 | 55.09 | 47.05 | 47.23 | 37.60 | 33.32 | 77.49 | 56.85 | 12.09 | 8.69 | | | Random Noise | 18.50 | 57.42 | 45.20 | 46.55 | 35.86 | 32.42 | 76.41 | 52.55 | 13.21 | 9.15 | | | Pitch Shifting | 20.55 | 59.32 | 46.80 | 48.17 | 37.22 | 34.34 | 78.17 | 54.50 | 12.93 | 10.04 | | | SpecAugment | 19.50 | 58.36 | 46.00 | 47.18 | 36.73 | 27.32 | 77.27 | 53.25 | 12.81 | 9.22 | | | Audiomentations | 20.35 | 60.13 | 47.25 | 48.30 | 38.24 | 28.15 | 79.12 | 54.51 | 13.28 | 11.44 | | | Retrieval | 19.20 | 37.14 | 42.55 | 43.65 | 35.80 | 31.27 | 71.42 | 51.35 | 10.53 | 7.28 | | 50 | Vanilla Syn. Aug. | 40.75 | 63.54 | 55.35 | 47.23 | 41.50 | 33.17 | 78.37 | 54.10 | 15.89 | 11.76 | | | + LLM Caps. | 36.80 | 65.84 | 63.74 | 55.36 | 40.90 | 38.17 | 78.77 | 57.05 | 13.07 | 11.84 | | | Synthio (ours) | 49.50 +122% | 76.12 +38% | 68.20 +44% | 60.58 +28% | 43.84 +17% | 40.83 +22% | 80.67 +4% | 60.15 +5% | 17.23 +42% | 15.41 +77% | | | w/ Template | 41.25 | 66.11 | 64.40 | 54.52 | 41.37 | 37.52 | 78.57 | 59.60 | 14.15 | 13.76 | | | Captions w/ ERM | 41.30 | 69.80 | 61.70 | 56.60 | 42.00 | 38.62 | 79.75 | 57.75 | 13.28 | 14.17 | | | w/o Self-Reflection | 45.25 | 72.57 | 64.55 | 58.00 | 42.81 | 39.50 | 78.56 | 57.25 | 15.63 | 14.02 | | | w/o MixCap | 42.70 | 64.72 | 54.65 | 52.18 | 41.93 | 36.13 | 78.70 | 58.80 | 14.82 | 13.61 | | | w/o DPO | 36.55 | 68.12 | 56.10 | 52.55 | 41.39 | 40.31 | 79.03 | 57.55 | 14.53 | 11.28 | | | Gold-only (No Aug.) | 56.75 | 72.89 | 64.15 | 57.81 | 47.14 | 39.11 | 84.32 | 65.60 | 12.50 | 10.70 | | | Random Noise | 58.50 | 71.54 | 65.50 | 56.98 | 46.21 | 38.20 | 83.33 | 66.15 | 13.35 | 14.95 | | | Pitch Shifting | 59.55 | 73.52 | 66.75 | 58.46 | 47.50 | 39.53 | 85.07 | 68.25 | 12.19 | 15.02 | | | SpecAugment | 47.50 | 72.43 | 69.75 | 58.06 | 50.07 | 41.96 | 85.14 | 66.40 | 14.17 | 15.72 | | | Audiomentations | 48.50 | 73.82 | 71.05 | 59.32 | 51.14 | 42.15 | 85.24 | 68.40 | 16.93 | 14.29 | | | Retrieval | 52.45 | 68.24 | 61.55 | 54.83 | 45.39 | 37.84 | 83.27 | 58.55 | 10.93 | 10.05 | | 100 | Vanilla Syn. Aug. | 77.25 | 77.31 | 68.25 | 63.58 | 49.96 | 42.31 | 84.78 | 63.55 | 15.73 | 13.78 | | | + LLM Caps. | 67.05 | 79.73 | 67.90 | 65.79 | 48.63 | 41.83 | 84.83 | 65.95 | 16.32 19.38 +55% | 14.12 17.33 +62% | | | Synthio (ours) | 83.35 +47% 78.00 | 85.00 +17% 80.32 | 71.20 +11% 68.15 | 71.23 +23% 64.20 | 52.42 +11% 49.95 | 44.92 +15% 42.76 | 86.70 +3% 85.11 | 68.80 +5% | | 14.32 | | | w/ Template Captions w/ ERM | 73.20 | 81.81 | 67.25 | 66.57 | 51.11 | 43.74 | 84.73 | 66.05 | 16.32 | | | | w/o Self-Reflection | | | | | | | | 68.00 | 17.21 | 15.19 | | | | 77.65 | 82.38 | 69.55 | 68.52 | 51.75 | 44.38 | 82.53 | 66.20 | 15.89 | 13.96 | | | w/o MixCap | 73.50 | 78.30 | 68.50 | 66.52 | 50.63 | 42.27 | 83.52 | 66.35 | 16.77 | 14.93 | | | w/o DPO Gold-only (No Aug.) | 66.75 | 75.46 74.80 | 66.15 77.00 | 60.81 67.41 | 48.78 55.32 | 40.31 48.77 | 84.67 87.38 | 67.85 68.80 | 14.83 23.15 | 13.92 14.82 | | | Random Noise | 84.75 | 75.15 | 75.50 | 66.71 | 54.42 | 47.83 | 86.45 | 65.45 | 24.82 | 17.77 | | | Pitch Shifting | 83.55 84.90 | 74.48 | 78.55 | 67.74 | 55.44 | 48.12 | 87.47 | 69.80 | 23.11 | 18.73 | | | SpecAugment | 85.10 | 76.46 | 76.25 | 65.70 | 55.72 | 54.80 | 87.42 | 69.25 | 27.36 | 18.91 | | | Audiomentations | 85.25 | 75.80 | 77.30 | 67.00 | 55.21 | 53.15 | 86.08 | 70.50 | 26.29 | 19.17 | | | Retrieval | 82.55 | 71.20 | 73.65 | 65.80 | 53.25 | 47.63 | 86.28 | 63.55 | 19.51 | 15.36 | | 200 | + LLM Caps. | 85.80 | 78.37 | 79.55 | 74.14 | 54.73 | 56.21 | 87.02 | 73.16 | 28.40 | | | | Vanilla Syn. Aug. | 85.40 | 77.96 | 77.10 | 78.97 | 55.51 | 55.20 | 86.49 | 72.95 | 28.55 | 19.04 19.34 | | | Synthio (ours) Captions | 86.10 +2% 85.95 | 82.81 +11% | 82.05 +7% | 79.40 +18% | 56.83 +3% | 57.10 +17% | 87.52 +0.2% 87.25 | 80.40 +17% 74.55 | 32.81 +42% 29.12 | 21.60 +46% 20.19 | | | w/ Template w/ ERM | 85.35 | 80.84 79.82 | 79.25 80.20 | 77.56 74.43 | 55.99 55.76 | 56.33 56.15 | 86.92 | 74.40 | 29.81 | 19.26 | | | w/o Self-Reflection | 84.85 | 81.97 | 78.25 | 75.53 | 56.39 | 56.76 | 86.22 | 75.55 | 31.13 | 18.45 | | | w/o MixCap | 84.95 | 81.27 | 79.55 | | | 55.54 | 85.78 | | | | | | w/o DPO | 84.80 | 76.23 | 75.30 | 73.50 73.13 | 55.27 55.99 | 52.73 | 86.52 | 78.55 73.15 | 28.35 26.79 | 20.58 19.58 | | | Gold-only (No Aug.) | 90.75 | 87.88 | 79.25 | 75.65 | 65.72 | 63.47 | 89.33 | 72.05 | 36.68 | 20.19 | | | Random Noise | 89.55 | 88.25 | 78.90 | 76.01 | 65.10 | 64.15 | 90.15 | 73.25 | 37.21 | 21.57 | | | | 88.50 | 88.83 | 79.75 | 75.61 | 64.93 | 64.59 | 89.87 | 72.15 | 36.54 | 22.17 | | | Pitch Shifting | 89.50 | 89.01 88.75 | 80.25 81.25 | 76.68 | 66.74 | 64.43 | 90.38 | 72.95 | 38.33 | 22.52 | | | SpecAugment Audiomentations | 89.95 | | | 77.66 | 66.92 | 65.21 | 91.34 | 73.65 | 38.75 | 23.11 | | 500 | Retrieval | 85.50 | 84.86 | 77.25 | 73.62 | 62.73 | 61.44 | 87.33 | 70.20 | 30.17 | | | | Aug. | | | | | | | | 73.25 | 37.26 | 15.28 24.39 | | | Vanilla Syn. | 91.50 | 88.18 | 79.35 79.55 | 77.97 77.91 | 65.93 | 64.52 | 90.31 | 73.05 | 38.74 | 23.83 24.93 | | | + LLM Caps. | 89.90 | 86.91 89.18 +2% | 82.25 +4% | 78.62 +4% | 65.95 | 64.39 | 90.09 91.42 | 74.70 +3% 73.35 | 39.24 +6% 38.28 | +23% 23.65 | | | Synthio (ours) w/ Template Captions | 92.10 +2% 91.70 | 88.93 88.25 | 80.40 | 76.64 | 67.81 +3% 66.47 | 65.40 +3% 64.71 | +2% 90.97 | 74.20 | | | | | w/ ERM | 91.20 91.85 | 88.72 | 79.15 80.15 | 77.38 78.57 | 65.80 66.21 | 64.27 63.89 | 88.74 90.17 | 72.15 | 38.03 37.97 | 23.72 23.62 | | | w/o Self-Reflection w/o MixCap | | 87.93 | | 76.61 | 65.91 | 64.23 | 90.23 | 73.40 | 39.11 | 22.83 | | | w/o DPO | 91.70 | | 80.95 | 76.03 | 66.01 | 63.61 | 89.83 | 72.65 | | | | | | 90.15 | 88.21 | 79.45 | | | | | | 37.04 | 21.28 | Main Results. Table 1 showcases the performance comparison between Synthio and the baseline methods. Synthio consistently outperforms all baselines by 0.1%-39%, achieving notable improvements in overall classification accuracy compared to Gold-only. The highest gains are observed on USD8K, while the least is on Vocal Sound, likely due to the T2A dataset's heavy representation of music compared to the more sparse vocal sounds. Performance gains tend to decrease as the number of gold samples n in D small grows, aligning with observed trends in prior studies. Although Vanilla Synthetic Augmentations emerge as the strongest baseline, they lag behind Synthio by 3.5%. Ablations. The most significant performance drop in Synthio is observed w/o DPO, resulting in an average decline of 4.5%, highlighting the crucial role of consistency in generating effective augmentations. Second to w/o DPO, the the highest drop is seen in w/ Template Captions, with average decline of 2.7%, thus highlighting the importance of MixCap. Figure 4: Comparison of spectral and pitch features between generated audios in D syn and real audios in D small (for n = 100). Synthio-generated audios closely replicate the features of real data, demonstrating its ability to produce augmentations that maintain consistency with the original dataset (also see FAD scores in Sec. A.4.3). <details> <summary>Image 4 Details</summary>  ### Visual Description ## Density Plots: Comparison of Audio Synthesis Methods Across Metrics ### Overview The image contains eight density plots comparing five audio synthesis methods (Gold, Synthia, Vanilla Syn. Avg., Synthia w/ Temp. Caps., Synthia w/ ERM) across four audio metrics (NSynth Pitch Salience, Spectral Complexity, Spectral Flatness, Spectral Flux) for two datasets (NSynth and USD8K). Each subplot visualizes the distribution of a metric's values across methods. --- ### Components/Axes - **X-Axes**: - NSynth Pitch Salience: -0.5 to 1.5 - Spectral Complexity: 10 to 45 (NSynth) / 10 to 27.5 (USD8K) - Spectral Flatness: -0.5 to 1.5 - Spectral Flux: -0.8 to 1.2 - **Y-Axes**: Density (scaled differently per subplot) - **Legends**: Located in the top-right of each subplot, with color-coded labels for methods. - **Datasets**: - Top row: NSynth - Bottom row: USD8K --- ### Detailed Analysis #### NSynth Metrics 1. **Pitch Salience** - **Gold** (blue): Peak at ~0.5, narrow distribution. - **Synthia** (orange): Peak at ~0.4, slightly broader. - **Vanilla Syn. Avg.** (green): Broad, flat distribution. - **Synthia w/ Temp. Caps.** (red): Narrow peak at ~0.3. - **Synthia w/ ERM** (purple): Broad, multimodal distribution. 2. **Spectral Complexity** - **Gold**: Peak at ~20, moderate spread. - **Synthia**: Peak at ~25, wider than Gold. - **Vanilla Syn. Avg.**: Peak at ~15, narrow. - **Synthia w/ Temp. Caps.**: Peak at ~30, sharp. - **Synthia w/ ERM**: Peak at ~20, overlaps with Gold. 3. **Spectral Flatness** - **Gold**: Peak near 0, sharp. - **Synthia**: Peak at ~0.2, moderate spread. - **Vanilla Syn. Avg.**: Peak at ~0.1, narrow. - **Synthia w/ Temp. Caps.**: Peak at ~0.3, broad. - **Synthia w/ ERM**: Peak at ~0.1, overlaps with Vanilla. 4. **Spectral Flux** - **Gold**: Peak at 0, narrow. - **Synthia**: Peak at ~0.1, moderate. - **Vanilla Syn. Avg.**: Peak at ~0.05, narrow. - **Synthia w/ Temp. Caps.**: Peak at ~0.2, broad. - **Synthia w/ ERM**: Peak at ~0.05, overlaps with Vanilla. #### USD8K Metrics 1. **Pitch Salience** - **Gold**: Peak at ~0.5, narrow. - **Synthia**: Peak at ~0.3, moderate. - **Vanilla Syn. Avg.**: Peak at ~0.4, broad. - **Synthia w/ Temp. Caps.**: Peak at ~0.2, narrow. - **Synthia w/ ERM**: Peak at ~0.3, overlaps with Synthia. 2. **Spectral Complexity** - **Gold**: Peak at ~15, narrow. - **Synthia**: Peak at ~17, moderate. - **Vanilla Syn. Avg.**: Peak at ~12, narrow. - **Synthia w/ Temp. Caps.**: Peak at ~20, sharp. - **Synthia w/ ERM**: Peak at ~15, overlaps with Gold. 3. **Spectral Flatness** - **Gold**: Peak near 0, sharp. - **Synthia**: Peak at ~0.1, moderate. - **Vanilla Syn. Avg.**: Peak at ~0.05, narrow. - **Synthia w/ Temp. Caps.**: Peak at ~0.2, broad. - **Synthia w/ ERM**: Peak at ~0.05, overlaps with Vanilla. 4. **Spectral Flux** - **Gold**: Peak at 0, narrow. - **Synthia**: Peak at ~0.1, moderate. - **Vanilla Syn. Avg.**: Peak at ~0.05, narrow. - **Synthia w/ Temp. Caps.**: Peak at ~0.2, broad. - **Synthia w/ ERM**: Peak at ~0.05, overlaps with Vanilla. --- ### Key Observations 1. **Method-Specific Trends**: - **Synthia w/ ERM** consistently produces broader distributions, suggesting higher variability in generated audio. - **Synthia w/ Temp. Caps.** often has sharper, narrower peaks, indicating more consistent outputs. - **Vanilla Syn. Avg.** shows flatter distributions, implying less variability. 2. **Dataset Differences**: - USD8K metrics generally have lower peak densities (e.g., Spectral Complexity peaks at ~15 vs. NSynth’s ~20). - Spectral Flux in USD8K has a y-axis scaled up to 2.0, unlike NSynth’s 0.5, indicating differing data ranges. 3. **Outliers**: - Synthia w/ Temp. Caps. in USD8K Spectral Complexity peaks at ~20, significantly higher than other methods. - Synthia w/ ERM in NSynth Spectral Flux overlaps with Vanilla Syn. Avg., suggesting similar performance. --- ### Interpretation The plots reveal how different synthesis methods affect audio characteristics: - **ERM (Empirical Risk Minimization)** introduces variability, as seen in broader distributions for Synthia w/ ERM. - **Temperature Caps.** likely stabilize outputs, resulting in sharper peaks (e.g., Synthia w/ Temp. Caps. in Spectral Complexity). - **Gold** (real data) serves as a benchmark, with Synthia methods sometimes aligning closely (e.g., Spectral Complexity in USD8K) or diverging (e.g., Spectral Flux in NSynth). The dataset-specific differences (NSynth vs. USD8K) highlight how synthesis methods perform across distinct audio domains. For instance, USD8K’s lower Spectral Complexity peaks may reflect simpler audio structures compared to NSynth. These insights suggest that method choice (e.g., ERM vs. Temp. Caps.) and dataset characteristics jointly influence audio quality and consistency. </details> ## 6.1 HOW CONSISTENT AND DIVERSE ARE AUGMENTATIONS GENERATED BY SYNTHIO? Fig. 4 compares the distributions of pitch and various spectral features between generated audios in D syn and real audios in D small across different methods on the USD8K and NSynth datasets. The features analyzed include Pitch Salience (clarity of the main pitch) (Ricard, 2004), Spectral Flatness (tonal vs. noise-like quality) (Peeters, 2004), Flux (rate of spectral change) (Tzanetakis & Cook, 1999), and Complexity (level of sound detail) (Laurier et al., 2010). Notably, Synthio-generated audios closely replicate the spectral features of the original audios, showing the best alignment among all methods and demonstrating Synthio's ability to generate consistent augmen- Table 2: CLAP similarity score between real audios and generated data. Lower scores show higher compositional diversity among generated augs. | Method | USD8K( ↓ ) | NSynth( ↓ ) | |----------------------|--------------|---------------| | Vanilla Syn. Aug. | 45.17 | 31.76 | | Synthio (ours) | 35.09 | 22.97 | | w/ Template Captions | 46.82 | 33 | | w/ ERM | 50.01 | 42.33 | | Vanilla Syn. Aug. | 47.22 | 33.81 | | Synthio (ours) | 34.58 | 23.03 | | w/ Template Captions | 46.84 | 37.16 | | w/ ERM | 52.54 | 43.98 | tations. Table 2 presents CLAP similarity scores between ground-truth audios and their N generated augmentations, averaged across all dataset instances. Audios generated with Synthio achieve the highest compositional diversity for generated audios among all baselines. Table 8 shows that audios generated using Synthio have the highest similarity with the ground-truth category label. ## 6.2 HOW GOOD ARE SYNTHETIC AUDIOS GENERATED BY SYNTHIO? Consistent with prior findings in vision (He et al., 2023), we observe that synthetic data alone performs sub-optimally compared to human-annotated data. However, our results show that enhancing the consistency and diversity of synthetic data aided by a smallscale version of the target dataset significantly improves model performance. Table 3 compares models trained exclusively on synthetic data with our baselines (i.e., only D syn is used for training AST). Synthio outperforms all baselines by 0.1%-26.25%, with DPO-based alignment driving the improvements. Table 3: Performance comparison of Synthio with baselines on synthetic-only audio classification. | Method | GTZAN | VS | TUT | MSDB | |----------------------|---------|-------|-------|--------| | Gold-only (No Aug.) | 64.15 | 84.32 | 47.14 | 65.6 | | Vanilla Syn. Aug. | 29.05 | 34.13 | 21.69 | 35.6 | | Synthio (ours) | 33.1 | 39.2 | 24.51 | 56.45 | | w/ Template Captions | 24.5 | 30.99 | 21.73 | 40.4 | | w/ ERM | 25.65 | 32.76 | 24.4 | 42.85 | | w/o DPO | 17.6 | 21.57 | 20.39 | 30.2 | | Gold-only (No Aug.) | 77 | 87.38 | 55.32 | 68.8 | | Vanilla Syn. Aug. | 32.35 | 41.96 | 24.23 | 39.25 | | Synthio (ours) | 35.15 | 48.14 | 27 | 61.45 | | w/ Template Captions | 29.9 | 35.53 | 23.61 | 41.2 | | w/ ERM | 28.1 | 36.29 | 25.71 | 46.7 | | w/o DPO | 19.85 | 26.85 | 21.4 | 36.75 | ## 6.3 CAN SYNTHIO BE EXTENDED TO THE MORE COMPLEX AUDIO CAPTIONING TASK? Audio captioning, unlike classification, involves describing the content of an audio sample using natural language, making it a more complex task. To demonstrate Synthio's effectiveness for audio captioning, we evaluated it on down-sampled versions of AudioCaps. For this task, we adapted Synthio by extracting components directly from the available captions and retrained our T2A model on a modified version of Sound-VECaps, excluding any audio from AudioSet. Training and evaluation were conducted using the EnCLAP framework (Kim et al., 2024), and the dataset was enriched with 4× synthetic samples. As shown in Table 4, Synthio significantly outperforms baseline settings, with improvements largely due to better alignment w/ DPO. However, manual inspection revealed that generated audios occasionally do not match their captions compositionally, reflecting limi- Table 4: Performance comparison of Synthio with baselines on audio captioning. | n | Method | METEOR ( ↑ ) | CIDEr ( ↑ ) SPICE ( ↑ | SPIDEr ( ↑ ) | |------------------------|----------|----------------|-------------------------|----------------| | Gold-only (No Aug.) | 0.125 | 0.148 | 0.0754 | 0.112 | | 500 Vanilla Syn. Aug. | 0.128 | 0.157 | 0.0741 | 0.136 | | VECaps Retrieval | 0.108 | 0.094 | 0.055 | 0.082 | | Synthio (ours) | 0.169 | 0.227 | 0.104 | 0.194 | | Gold-only (No Aug.) | 0.127 | 0.157 | 0.067 | 0.112 | | 1000 Vanilla Syn. Aug. | 0.135 | 0.166 | 0.092 | 0.14 | | VECaps Retrieval | 0.088 | 0.097 | 0.068 | 0.1 | | Synthio (ours) | 0.185 | 0.256 | 0.119 | 0.202 | tations of the current T2A model. While this issue does not affect classification, it poses challenges for captioning. We will explore more advanced methods as part of future work. ## 6.4 HOW WELL DOES SYNTHIO SCALE? Table 5 compares the performance of Synthio, SpecAugment, and Vanilla Synthetic Augmentations across different scaling factors N = { 1, 2, 3, 4, 5 } , where N represents the number of synthetic samples generated per original sample in the small-scale dataset. As observed, SpecAugment, a traditional augmentation method, cannot scale with increasing N , and the performance of Vanilla Synthetic Augmentations plateaus at higher N . Asimilar saturation occurs with Synthio when MixCap is not used. Even without DPO, Synthio maintains better scalability, though with reduced overall performance. These re- Table 5: Performance comparison of Synthio with other baselines on different values of N . | Dataset | Method | Scaling Factor N | Scaling Factor N | Scaling Factor N | Scaling Factor N | Scaling Factor N | |-----------|-------------------|--------------------|--------------------|--------------------|--------------------|--------------------| | Dataset | Method | 1x | 2x | 3x | 4x | 5x | | ESC50 | SpecAugment | 47.50 | 47.50 | 47.50 | 47.50 | 47.50 | | ESC50 | Vanilla Syn. Aug. | 67.90 | 77.25 | 76.75 | 75.60 | 71.25 | | ESC50 | Synthio (ours) | 77.45 | 81.75 | 82.55 | 83.15 | 83.35 | | ESC50 | w/o MixCap | 64.30 | 68.45 | 71.55 | 72.85 | 73.50 | | ESC50 | w/o DPO | 61.55 | 64.25 | 65.95 | 66.60 | 66.75 | | NSynth | SpecAugment | 41.96 | 41.96 | 41.96 | 41.96 | 41.96 | | NSynth | Vanilla Syn. Aug. | 33.13 | 35.28 | 42.31 | 41.54 | 38.27 | | NSynth | Synthio (ours) | 35.28 | 36.37 | 43.56 | 44.92 | 44.81 | | NSynth | w/o MixCap | 40.41 | 41.08 | 41.95 | 42.27 | 42.15 | | NSynth | w/o DPO | 39.23 | 39.42 | 40.17 | 40.31 | 39.82 | sults highlight that MixCap's ability to generate diverse captions is crucial for Synthio's scalability. ## 6.5 DOES SYNTHIO HELP LONG-TAILED CATEGORIES? Figure 5 shows the classification accuracy on four underrepresented categories in the ESC-50 and NSynth datasets, comparing performance before and after applying Synthio augmentations. We selected categories with the lowest frequency in the datasets, such as flute and guitar , which appear only once in the down-sampled sets. Synthio significantly boosts accuracy, with improvements up to 48%. Notably, categorie labels like flute and guitar , which originally had 0% accuracy, show substantial gains with Synthio augmentation. This demonstrates Figure 5: Category-wise improvement in performance with Synthio augmentations for long-tailed categories. <details> <summary>Image 5 Details</summary>  ### Visual Description ## Bar Chart: Classification Accuracy Comparison (Synthio vs Gold-only) ### Overview The chart compares classification accuracy percentages between two methods ("Improvement w/ Synthio" and "Gold-only") across eight sound categories. The y-axis represents accuracy in percentage (0–100%), and the x-axis lists categories: street music, air conditioner, car horn, gun shot, flute, guitar, reed, and mallet. Two bars per category are shown: green for "Improvement w/ Synthio" and red for "Gold-only". ### Components/Axes - **X-axis (Categories)**: Street music, air conditioner, car horn, gun shot, flute, guitar, reed, mallet. - **Y-axis (Classification Accuracy)**: 0–100% scale. - **Legend**: - Green: "Improvement w/ Synthio" - Red: "Gold-only" - **Legend Placement**: Top-right corner. - **Bar Colors**: - Green bars represent "Improvement w/ Synthio". - Red bars represent "Gold-only". ### Detailed Analysis - **Street Music**: - Gold-only: 44.00% (red) - Improvement: 39.00% (green) - **Air Conditioner**: - Gold-only: 48.00% (red) - Improvement: 48.39% (green) - **Car Horn**: - Gold-only: 35.48% (red) - Improvement: 48.39% (green) - **Gun Shot**: - Gold-only: 68.57% (red) - Improvement: 31.43% (green) - **Flute**: - Gold-only: 21.11% (red) - Improvement: 14.60% (green) - **Guitar**: - Gold-only: 14.60% (red) - Improvement: 20.00% (green) - **Reed**: - Gold-only: 20.00% (red) - Improvement: 15.34% (green) - **Mallet**: - Gold-only: 64.36% (red) - Improvement: 15.34% (green) ### Key Observations 1. **Mixed Performance**: - "Improvement w/ Synthio" outperforms "Gold-only" in **car horn** (+12.91%) and **guitar** (+5.40%). - "Improvement w/ Synthio" underperforms in **gun shot** (-37.14%), **flute** (-30.85%), and **mallet** (-76.67%). 2. **Highest Accuracy**: - "Gold-only" achieves the highest accuracy in **gun shot** (68.57%) and **mallet** (64.36%). 3. **Lowest Accuracy**: - "Improvement w/ Synthio" performs worst in **flute** (14.60%) and **mallet** (15.34%). ### Interpretation The data reveals that the Synthio method’s effectiveness is **category-dependent**. It shows significant improvement in **car horn** and **guitar** but degrades performance in **gun shot**, **flute**, and **mallet**. This suggests that Synthio’s synthetic data augmentation may introduce noise or misalignment in certain acoustic contexts (e.g., complex sounds like gunshots or mallets). The "Gold-only" method remains more reliable for high-accuracy tasks in challenging categories, while Synthio offers marginal gains in simpler or mid-range scenarios. The stark drop in mallet accuracy highlights potential overfitting or dataset imbalance issues when using synthetic data for rare or complex sounds. </details> Synthio's effectiveness in enhancing performance on long-tail labels, a common challenge in realworld datasets (Zhang et al., 2023). ## 7 CONCLUSION, LIMITATIONS, AND FUTURE WORK We introduced Synthio, a novel approach for augmenting small-scale audio classification datasets with synthetic data. Synthio incorporates several innovative components to generate augmentations that are both consistent with and diverse from the small-scale dataset. Our extensive experiments demonstrate that even when using a T2A model trained on a weakly-captioned AudioSet, Synthio significantly outperforms multiple baselines. However, Synthio has some limitations: (i) Its performance is influenced by the capabilities of the T2A model and the quality of its training data. As T2A models continue to improve, we expect Synthio's performance to benefit accordingly. (ii) The process of generating audio captions using LLMs may introduce biases inherent in the LLMs into the training process. (iii) Synthio is computationally more intensive than traditional augmentation methods due to the need for prompting LLMs and T2A models. We anticipate that ongoing advancements in model efficiency will help mitigate these computational challenges. ## 8 REPRODUCIBILITY STATEMENT All codes will be open-sourced upon paper acceptance, including all T2A checkpoints. All experimental details, including training parameters and hyper-parameters, are provided in Section 5. ## REFERENCES - Haider Al-Tahan and Yalda Mohsenzadeh. Clar: Contrastive learning of auditory representations. In International Conference on Artificial Intelligence and Statistics , pp. 2530-2538. PMLR, 2021. - Max F Burg, Florian Wenzel, Dominik Zietlow, Max Horn, Osama Makansi, Francesco Locatello, and Chris Russell. Image retrieval outperforms diffusion models on data augmentation. Transactions on Machine Learning Research , 2023. ISSN 2835-8856. URL https://openreview.net/ forum?id=xflYdGZMpv . - Jiaao Chen, Derek Tam, Colin Raffel, Mohit Bansal, and Diyi Yang. An empirical survey of data augmentation for limited data learning in nlp. Transactions of the Association for Computational Linguistics , 11:191-211, 2021. URL https://api.semanticscholar.org/CorpusID: 235422524 . - Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems , 30, 2017. - Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre D´ efossez. Simple and controllable music generation. Advances in Neural Information Processing Systems , 36, 2024. - Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen. Clotho: An audio captioning dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pp. 736-740. IEEE, 2020. - Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. Clap learning audio concepts from natural language supervision. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pp. 1-5. IEEE, 2023. - Zach Evans, CJ Carr, Josiah Taylor, Scott H Hawley, and Jordi Pons. Fast timing-conditioned latent audio diffusion. arXiv preprint arXiv:2402.04825 , 2024a. - Zach Evans, Julian D Parker, CJ Carr, Zack Zukowski, Josiah Taylor, and Jordi Pons. Stable audio open. arXiv preprint arXiv:2407.14358 , 2024b. - Eduardo Fonseca, Xavier Favory, Jordi Pons, Frederic Font, and Xavier Serra. FSD50K: an open dataset of human-labeled sound events. IEEE/ACM Transactions on Audio, Speech, and Language Processing , 30:829-852, 2022. - Jonas Geiping, Micah Goldblum, Gowthami Somepalli, Ravid Shwartz-Ziv, Tom Goldstein, and Andrew Gordon Wilson. How much data are augmentations worth? an investigation into scaling laws, invariance, and implicit regularization. arXiv preprint arXiv:2210.06441 , 2022. - Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP) , pp. 776-780. IEEE, 2017. - Deepanway Ghosal, Navonil Majumder, Ambuj Mehrish, and Soujanya Poria. Text-to-audio generation using instruction tuned llm and latent diffusion model. arXiv preprint arXiv:2304.13731 , 2023. - Sreyan Ghosh, Ashish Seth, and S Umesh. Decorrelating feature spaces for learning general-purpose audio representations. IEEE Journal of Selected Topics in Signal Processing , 16(6):1402-1414, 2022. doi: 10.1109/JSTSP.2022.3202093. - Sreyan Ghosh, Chandra Kiran Reddy Evuru, Sonal Kumar, S Ramaneswaran, S Sakshi, Utkarsh Tyagi, and Dinesh Manocha. DALE: Generative data augmentation for low-resource legal NLP. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pp. 8511-8565, Singapore, December 2023a. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.528. URL https://aclanthology.org/2023.emnlp-main.528 . - Sreyan Ghosh, Ashish Seth, Srinivasan Umesh, and Dinesh Manocha. Mast: Multiscale audio spectrogram transformers. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pp. 1-5. IEEE, 2023b. - Sreyan Ghosh, Sonal Kumar, Ashish Seth, Chandra Kiran Reddy Evuru, Utkarsh Tyagi, S Sakshi, Oriol Nieto, Ramani Duraiswami, and Dinesh Manocha. Gama: A large audio-language model with advanced audio understanding and complex reasoning abilities, 2024a. URL https:// arxiv.org/abs/2406.11768 . - Sreyan Ghosh, Ashish Seth, Sonal Kumar, Utkarsh Tyagi, Chandra Kiran Reddy Evuru, Ramaneswaran S, S Sakshi, Oriol Nieto, Ramani Duraiswami, and Dinesh Manocha. Compa: Addressing the gap in compositional reasoning in audio-language models. In The Twelfth International Conference on Learning Representations , 2024b. URL https://openreview.net/ forum?id=86NGO8qeWs . - Sreyan Ghosh, Utkarsh Tyagi, Sonal Kumar, Chandra Kiran Reddy Evuru, , Ramaneswaran S, S Sakshi, and Dinesh Manocha. ABEX: Data augmentation for low-resource NLU via expanding abstract descriptions. In The 62nd Annual Meeting of the Association for Computational Linguistics , 2024c. - Yuan Gong, Yu-An Chung, and James Glass. Ast: Audio spectrogram transformer. arXiv preprint arXiv:2104.01778 , 2021. - Ruifei He, Shuyang Sun, Xin Yu, Chuhui Xue, Wenqing Zhang, Philip Torr, Song Bai, and XIAOJUAN QI. IS SYNTHETIC DATA FROM GENERATIVE MODELS READY FOR IMAGE RECOGNITION? In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=nUmCcZ5RKF . - I Jordal. audiomentations, 2021. URL https://zenodo.org/record/13639627 . - Akbar Karimi, Leonardo Rossi, and Andrea Prati. AEDA: An easier data augmentation technique for text classification. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wentau Yih (eds.), Findings of the Association for Computational Linguistics: EMNLP 2021 , pp. 2748-2754, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.findings-emnlp.234. URL https://aclanthology. org/2021.findings-emnlp.234 . - Kevin Kilgour, Mauricio Zuluaga, Dominik Roblek, and Matthew Sharifi. Fr \ 'echet audio distance: A metric for evaluating music enhancement algorithms. arXiv preprint arXiv:1812.08466 , 2018. - Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. Audiocaps: Generating captions for audios in the wild. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pp. 119-132, 2019. - Jaeyeon Kim, Jaeyoon Jung, Jinjoo Lee, and Sang Hoon Woo. Enclap: Combining neural audio codec and audio-text joint embedding for automated audio captioning. In ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pp. 6735-6739, 2024. doi: 10.1109/ICASSP48485.2024.10446672. - Zhifeng Kong, Sang-gil Lee, Deepanway Ghosal, Navonil Majumder, Ambuj Mehrish, Rafael Valle, Soujanya Poria, and Bryan Catanzaro. Improving text-to-audio models with synthetic captions. arXiv preprint arXiv:2406.15487 , 2024. - Felix Kreuk, Gabriel Synnaeve, Adam Polyak, Uriel Singer, Alexandre D´ efossez, Jade Copet, Devi Parikh, Yaniv Taigman, and Yossi Adi. Audiogen: Textually guided audio generation. In The Eleventh International Conference on Learning Representations , 2023. URL https: //openreview.net/forum?id=CYK7RfcOzQ4 . - Cyril Laurier, Owen Meyers, Joan Serra, Martin Blech, Perfecto Herrera, and Xavier Serra. Indexing music by mood: design and integration of an automatic content-based annotator. Multimedia Tools and Applications , 48:161-184, 2010. - Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. Audioldm: Text-to-audio generation with latent diffusion models. arXiv preprint arXiv:2301.12503 , 2023. - Lin Long, Rui Wang, Ruixuan Xiao, Junbo Zhao, Xiao Ding, Gang Chen, and Haobo Wang. On llms-driven synthetic data generation, curation, and evaluation: A survey. arXiv preprint arXiv:2406.15126 , 2024. - Vincent Lostanlen and Carmine-Emanuele Cella. Deep convolutional networks on the pitch spiral for musical instrument recognition, 2017. URL https://arxiv.org/abs/1605.06644 . - Navonil Majumder, Chia-Yu Hung, Deepanway Ghosal, Wei-Ning Hsu, Rada Mihalcea, and Soujanya Poria. Tango 2: Aligning diffusion-based text-to-audio generations through direct preference optimization. arXiv preprint arXiv:2404.09956 , 2024. - Pranay Manocha, Zeyu Jin, Richard Zhang, and Adam Finkelstein. Cdpam: Contrastive learning for perceptual audio similarity. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pp. 196-200. IEEE, 2021. - Irene Mart´ ın-Morat´ o and Annamaria Mesaros. What is the ground truth? reliability of multi-annotator data for audio tagging. In 2021 29th European Signal Processing Conference (EUSIPCO) , pp. 76-80. IEEE, 2021. - Annamaria Mesaros, Toni Heittola, Aleksandr Diment, Benjamin Elizalde, Ankit Shah, Emmanuel Vincent, Bhiksha Raj, and Tuomas Virtanen. Dcase 2017 challenge setup: Tasks, datasets and baseline system. In DCASE 2017-Workshop on Detection and Classification of Acoustic Scenes and Events , 2017. - Loris Nanni, Gianluca Maguolo, and Michelangelo Paci. Data augmentation approaches for improving animal audio classification. Ecological Informatics , 57:101084, 2020. - An Thanh Nguyen, Byron Wallace, Junyi Jessy Li, Ani Nenkova, and Matthew Lease. Aggregating and predicting sequence labels from crowd annotations. In Regina Barzilay and Min-Yen Kan (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp. 299-309, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653/v1/P17-1028. URL https://aclanthology.org/P17-1028 . - Daisuke Niizumi, Daiki Takeuchi, Yasunori Ohishi, Noboru Harada, and Kunio Kashino. Byol for audio: Self-supervised learning for general-purpose audio representation. In 2021 International Joint Conference on Neural Networks (IJCNN) , pp. 1-8. IEEE, 2021. - Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le. Specaugment: A simple data augmentation method for automatic speech recognition. arXiv preprint arXiv:1904.08779 , 2019. - Geoffroy Peeters. A large set of audio features for sound description (similarity and classification) in the cuidado project. CUIDADO Ist Project Report , 54(0):1-25, 2004. - Karol J. Piczak. ESC: Dataset for Environmental Sound Classification. In Proceedings of the 23rd Annual ACM Conference on Multimedia , pp. 1015-1018. ACM Press. ISBN 978-1-4503-34594. doi: 10.1145/2733373.2806390. URL http://dl.acm.org/citation.cfm?doid= 2733373.2806390 . - Zafar Rafii, Antoine Liutkus, Fabian-Robert St¨ oter, Stylianos Ioannis Mimilakis, and Rachel Bittner. The MUSDB18 corpus for music separation, December 2017. URL https://doi.org/10. 5281/zenodo.1117372 . - Zhao Ren, Kun Qian, Tanja Schultz, and Bj¨ orn W. Schuller. An overview of the icassp special session on ai security and privacy in speech and audio processing. In Proceedings of the 5th ACM International Conference on Multimedia in Asia Workshops , MMAsia '23 Workshops, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400703263. doi: 10.1145/3611380.3628563. URL https://doi.org/10.1145/3611380.3628563 . - Julien Ricard. Towards computational morphological description of sound. DEA pre-thesis research work, Universitat Pompeu Fabra, Barcelona , 2004. - Francesca Ronchini, Luca Comanducci, and Fabio Antonacci. Synthesizing soundscapes: Leveraging text-to-audio models for environmental sound classification. arXiv preprint arXiv:2403.17864 , 2024. - Aaqib Saeed, David Grangier, and Neil Zeghidour. Contrastive learning of general-purpose audio representations. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pp. 3875-3879. IEEE, 2021. - J. Salamon, C. Jacoby, and J. P. Bello. A dataset and taxonomy for urban sound research. In 22nd ACM International Conference on Multimedia (ACM-MM'14) , pp. 1041-1044, Orlando, FL, USA, Nov. 2014. - Ashish Seth, Sreyan Ghosh, Srinivasan Umesh, and Dinesh Manocha. Slicer: Learning universal audio representations using low-resource self-supervised pre-training. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pp. 1-5. IEEE, 2023. - Connor Shorten and Taghi M Khoshgoftaar. A survey on image data augmentation for deep learning. Journal of big data , 6(1):1-48, 2019. - Janne Spijkervet. Spijkervet/torchaudio-augmentations, 2021. URL https://zenodo.org/ record/4748582 . - Jinchuan Tian, Chunlei Zhang, Jiatong Shi, Hao Zhang, Jianwei Yu, Shinji Watanabe, and Dong Yu. Preference alignment improves language model-based tts. arXiv preprint arXiv:2409.12403 , 2024. - Brandon Trabucco, Kyle Doherty, Max A Gurinas, and Ruslan Salakhutdinov. Effective data augmentation with diffusion models. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id=ZWzUA9zeAg . - George Tzanetakis and Perry Cook. Multifeature audio segmentation for browsing and annotation. In Proceedings of the 1999 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics. WASPAA'99 (Cat. No. 99TH8452) , pp. 103-106. IEEE, 1999. - George Tzanetakis, Georg Essl, and Perry Cook. Automatic musical genre classification of audio signals, 2001. URL http://ismir2001.ismir.net/pdf/tzanetakis.pdf . - Jason Wang, Luis Perez, et al. The effectiveness of data augmentation in image classification using deep learning. Convolutional Neural Networks Vis. Recognit , 11(2017):1-8, 2017. - Yuanyuan Wang, Hangting Chen, Dongchao Yang, Zhiyong Wu, Helen Meng, and Xixin Wu. Audiocomposer: Towards fine-grained audio generation with natural language descriptions. arXiv preprint arXiv:2409.12560 , 2024. - Jason Wei and Kai Zou. EDA: Easy data augmentation techniques for boosting performance on text classification tasks. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pp. 63826388, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1670. URL https://aclanthology.org/D19-1670 . - Zeng Weili, Yichao Yan, Qi Zhu, Zhuo Chen, Pengzhi Chu, Weiming Zhao, and Xiaokang Yang. Infusion: Preventing customized text-to-image diffusion from overfitting. In ACM Multimedia 2024 , 2024. - Yi Yuan, Dongya Jia, Xiaobin Zhuang, Yuanzhe Chen, Zhengxi Liu, Zhuo Chen, Yuping Wang, Yuxuan Wang, Xubo Liu, Mark D Plumbley, et al. Improving audio generation with visual enhanced caption. arXiv preprint arXiv:2407.04416 , 2024. Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision , pp. 6023-6032, 2019. Yifan Zhang, Bingyi Kang, Bryan Hooi, Shuicheng Yan, and Jiashi Feng. Deep long-tailed learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence , 45(9):10795-10816, 2023. - Qihao Zhao, Yalun Dai, Hao Li, Wei Hu, Fan Zhang, and Jun Liu. Ltgc: Long-tail recognition via leveraging llms-driven generated content. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 19510-19520, June 2024. ## A APPENDIX ## Table of Contents: - A.1 Background on Diffusion Models - A.2 Prompts - A.3 Examples - A.4 Extra Results - A.5 Dataset Details ## A.1 DIFFUSION MODELS Diffusion models consist of two main processes: a forward process and a reverse process. Given a data point x 0 with probability distribution p ( x 0 ) , the forward diffusion process gradually adds Gaussian noise to x 0 according to a pre-set variance schedule β 1 , ⋯ , β T and degrades the structure of the data. At the time step t , the latent variable x t is only determined by the x t -1 due to its discrete-time Markov process nature, and can be expressed as: $$p ( x _ { 1 } | x _ { t - 1 } ) = N ( x _ { i } \sqrt { I - \beta _ { t x _ { t - 1 } } }$$ As t increases over several diffusion steps, p ( x T ) approaches a unit spherical Gaussian distribution. The marginal distribution of x t at any given step can be expressed analytically as: $$\rho ( x _ { t } | x _ { 0 } ) = N ( x _ { t } ; \sqrt { a _ { t } x _ { 0 } } , (1 - c )$$ where α t = ∏ t s = 1 ( 1 -β s ) . The reverse process aims to reconstruct the original data from the noise-corrupted version by learning a series of conditional distributions. The transition from x t to x t -1 is modeled as: $$\rho _ { 0 } ( x _ { t - 1 } | x _ { t } ) = N ( x _ { t - 1 } ; \mu ^ { t - 1 } _ { 0 } , s ^ { t - 1 } _ { 0 }$$ $$t _ { 0 } ^ { t - 1 } = \frac { 1 } { \sqrt { a _ { 1 } } ( x _ { 1 } - \frac { b _ { 1 } } { \sqrt { 1 - a _ { 1 } } } ) } ,$$ $$\sigma _ { 0 } ^ { - 1 } = \frac { 1 - a _ { t - 1 } } { 1 - a _ { t } } \cdot \beta _ { t }$$ where α t = 1 -β t , ¯ α t = ∏ t i = 1 α i , θ represents the learnable parameters, µ t -1 θ is the mean estimate, σ t -1 2 θ is the standard deviation estimate, and ϵ θ ( x t , t ) is the noise estimated by the neural network. The reverse process estimates the data distribution p ( x 0 ) by integrating over all possible paths: $$p _ { 0 } ( x _ { 0 } ) = \int p _ { 0 } ( x _ { T } ) \sqrt { p _ { 0 } ( x _ { - 1 } ) } d x _ { 1 }$$ where p θ ( x T ) = N ( x T ; 0 , I ) . At inference time, the diffusion model iteratively executes the reverse process (Eq. 17) T times starting from a randomly sampled Gaussian Noise ( ϵ ∼ N ( 0 , I )) . ## A.2 PROMPTS Fig. 6, 7, 8 and 9 illustrate all the prompts used in our experiments. For all experiments, we prompt GPT-4-Turbo (GPT-4-turbo-2024-04-09) with top-p=0.5 and temperature=0.7. Figure 6: LLM prompt (Prompt 1) for extracting components from audio captions. <details> <summary>Image 6 Details</summary>  ### Visual Description ## Text-Based Instruction Set: Audio Feature Extraction Protocol ### Overview The image contains a technical instruction set for extracting and categorizing audio features from captions and labels. It outlines a structured JSON output format with three keys: `events`, `scenes`, and `other features`, each requiring specific types of audio-related information. ### Components/Axes - **Keys**: 1. `events`: Unique foreground events in the caption. 2. `scenes`: Broader background scene or background events in the audio. 3. `other features`: Additional audio-related features not covered by the first two keys. - **Output Format**: - JSON structure with comma-separated Pythonic lists for each key. - Empty values if no information is available for a key. ### Detailed Analysis - **Text Content**: The text provides explicit instructions for parsing audio descriptions into three categories: - **Foreground Events**: Specific, unique occurrences in the audio (e.g., "a dog barking"). - **Background Scenes**: General environmental context (e.g., "city street noise"). - **Other Features**: Miscellaneous audio attributes (e.g., "echo effect"). - Placeholders for `caption` and `label` are included but left empty in the example. - **Spatial Grounding**: - The text is left-aligned with no visual hierarchy beyond paragraph breaks. - No legends, axes, or numerical data present. ### Key Observations - The instructions emphasize ignoring phrases that do not semantically align with the audio label. - No numerical values, trends, or data points are provided in the image. ### Interpretation This protocol is designed for natural language processing (NLP) tasks involving audio-visual data alignment. It prioritizes semantic relevance over exhaustive extraction, ensuring only contextually meaningful phrases are included. The absence of example data suggests this is a template for future use, requiring manual or automated input of actual audio descriptions. ## No numerical data, charts, or diagrams present. The image is purely textual. </details> Figure 7: LLM prompt (Prompt 2) for generating new audio captions given elements from existing captions. <details> <summary>Image 7 Details</summary>  ### Visual Description ## Text Block: Audio Caption Rewriting Instructions ### Overview The image contains a structured template for rewriting audio captions to train an audio classification model. It outlines rules for incorporating domain-specific features (events, backgrounds, other features) while preserving the original audio event label. ### Components/Axes - **Text Structure**: - **Header**: Instructions for caption rewriting (rules 1 and 2). - **Placeholders**: - `{label}`: Abstract audio description (e.g., "birdsong in a forest"). - `{domain}`: Contextual domain (e.g., "outdoor wildlife recording"). - `{events}`, `{backgrounds}`, `{other features}`: Lists of audio features to incorporate. - **Output Directive**: Rewritten caption or "None" if no changes are made. ### Detailed Analysis - **Rule 1**: - Rewrite the caption to include features from the provided lists (events, backgrounds, other features). - Ensure the rewritten caption is unique but inspired by the given features. - **Rule 2**: - The rewritten caption must still align with the original audio event label. - **Placeholders**: - All placeholders (`{label}`, `{domain}`, `{events}`, etc.) are empty in the example, indicating they are to be filled dynamically. ### Key Observations - The template enforces a balance between creativity (Rule 1) and fidelity to the original label (Rule 2). - Placeholders suggest modularity, allowing adaptation to different audio datasets. - No numerical data or visual elements (e.g., charts) are present. ### Interpretation This template is designed for **audio classification model training**, where captions must reflect both the specific audio content (e.g., "dog barking") and broader contextual features (e.g., "urban environment, daytime"). By requiring feature diversity while preserving the label, it ensures the model learns nuanced relationships between audio events and their descriptions. The absence of pre-filled placeholders implies this is a reusable framework, not a static dataset. **Note**: No charts, diagrams, or numerical data are present in the image. All information is textual and procedural. </details> ## A.3 EXAMPLES Table 6 presents examples of captions generated by the Synthio framework, along with their revised versions for captions that were initially rejected. I will provide you with a label for an audio. The label generally describes the audio in an abstract fashion and mentions the broader scene or event that I need to teach an audio model about from the audio, i.e., the audio and its label is part of the training set for training an audio classification model. I will also provide you with the domain of the audio which will help you identify the true sound conveyed in the label. I would like you to generate 5 new captions that describe the event or source in the label in diverse fashions. I will use these captions to generate new audios that can augment my training set. Generate the new captions with the following requirements: 1. All the captions need to include new and diverse events and contexts beyond the actual event conveyed by the label. 2. Only add new events and context by understanding the broader context of the occurrence of the audio and the target label. Do not add random events or contexts. 3. The new caption should be not more than 20-25 words. 4. However, after all these constraints and adding new events or contexts, the caption still needs to obey the event conveyed by the original label, i.e., the new caption may not lead to an audio generation that defies the audio label. 6. Finally, use the original label as a phrase in your caption. Here is the label:{}. Here is the domain of the audio:{}. Output a JSON with the key as the original label and a value as the list of comma separated new captions. Only output the JSON and nothing else Figure 8: LLM prompt for generating random captions for Random Captions baselines in Table 1. I will provide you with a label for an audio. The label generally describes the audio in an abstract fashion and mentions the broader scene or event that I need to teach an audio model about from the audio, i.e., the audio and its label is part of the training set for training an audio classification model. I will also provide you with the domain of the audio which will help you identify the true sound conveyed in the label. I would like you to generate 5 new captions that describe the event or source in the label in diverse fashions. I will use these captions to generate new audios that can augment my training set. Generate the new captions with the following requirements: 1. Each caption should have a diverse added events (beyond the event of the original label) and contexts. 2. Only add new events and context by understanding the broader context of the occurrence of the audio and the target label. For adding events and contexts, please follow the next requirement. 3. I will also provide you with a list of features extracted from an existing set of audios. You should try such that the new captions you generate for the label have a mix of events and scenes similar to the events and background scenes that are given and new scenes, i.e., you should try to describe a scene for the caption with the events and backgrounds provided to you in the given lists but you should also add novel features (events, backgrounds or other features) beyond the ones given. 4. The new caption should be not more than 20-25 words. 5. However, after all these constraints and adding new events or contexts, the caption still needs to obey the event label, i.e., the new caption may not lead to an audio generation that defies the audio label. 6. Finally, use the original label as a phrase in your caption. ``` Here is the label: (); Here is the domain of the audio:(); Here is the list of events:(); Here is the list of backgrounds:(); Here is the list of other features:(); ``` Output a JSON with the key as the original caption and a value as the list of comma separated new captions. Only output the JSON and nothing else. Figure 9: LLM prompt (Prompt 3) for rewriting captions of rejected audios. ## A.4 EXTRA RESULTS ## A.4.1 RESULTS ON THE FULL TRAINING SPLITS Table 7 presents the performance comparison of Synthio on the full original dataset splits (where the entire training set is used without any downsampling). While Synthio outperforms all baselines, traditional augmentation methods prove to be much more competitive in this scenario. This contrasts Table 6: Examples of generated and revised captions from the Synthio methodology. | Dataset | Label | Generated Caption | Revised Caption | |------------|------------------|------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------| | USD8k | children playing | Children playing in a bustling city park with distant traffic noise | NA | | USD8k | children playing | Children playing in a schoolyard during recess with teacher's whistle | Children playing in a neighborhood al- ley with sound of distant construction | | USD8k | street music | Street music playing near a busy in- tersection filled with honking cars and pedestrians. | NA | | USD8k | street music | Street music from a bustling market as people chatter and vendors shout | Street music echoing through an alley- way during a lively street festival. | | TUT | airport | airport with people talking and walking around in an empty hallway | NA | | TUT | airport | In the airport, people are talking with the sound of a crowd of people in the background, as announcements play. | airport ambiance with people talking and children running around | | TUT | bus | Bus passing by on a road while people are chatting at a nearby cafe. | NA | | TUT | bus | bus passing by on a road as it continues to blow into the microphone | bus idling on a road with birds chirping nearby | | NSynth | keyboard | keyboard accompaniment to a live band performance at a bustling cafe. | NA | | NSynth | keyboard | a man typing on a keyboard at office | keyboard rhythms echoing in an empty auditorium during a rehearsal break | | NSynth | organ | A serene church service with an organ playing a melody and soft brass are playing. | NA | | NSynth | organ | An organ plays as guitars are playing together in the background. | An organ plays during a lively music festival with various instruments. | | Medley | Violin | violin being played during a classical symphony orchestra performance | NA | | Medley | Violin | violin performing a lively jig at a bustling street fair | Violin solo during a quiet candlelight dinner in a fancy restaurant. | | Medley | Flute | flute playing in a tranquil forest during the early morning | NA | | Medley | Flute | Flute performance in a bustling city park during a sunny afternoon. | Flute music echoing in an ancient stone cathedral. | | AudioCaps- | AudioCaps- | A dog barks repeatedly in the back- ground while a car engine starts | - | | AudioCaps- | AudioCaps- | In the distance, a faint thunder rumble is audible, accompanied by the gentle rustling of leaves in the wind. | Soft rain falls on a metal roof, creating a rhythmic tapping sound. | Table 7: Comparison of Synthio and other baselines on the full original dataset splits (using all samples from the original training set as D small ). | Method | USD8K | GTZAN | Medley | VS | MSDB | |-------------------|---------|---------|----------|-------|--------| | Gold-only | 88.23 | 82 | 80.99 | 92.73 | 73.9 | | Random Noise | 86.17 | 82.35 | 79.72 | 92.94 | 74.55 | | Pitch Shift | 87.58 | 83.02 | 79.63 | 92.17 | 74.6 | | Spec. Aug. | 87.92 | 82.5 | 79.14 | 92.42 | 74.5 | | Audiomentations | 88.01 | 82.75 | 81.26 | 92.47 | 75.05 | | Retrieval | 78.27 | 69.25 | 73.24 | 80.43 | 69.95 | | Vanilla Syn. Aug. | 89.57 | 82.85 | 81.79 | 93.15 | 75.85 | | Synthio (ours) | 89.57 | 82.85 | 81.79 | 93.01 | 74.24 | with the results in Table 1 where traditional augmentations showed minimal improvements in performance. Table 8: CLAP score between generated audios and the label. | n | Method | USD8K | NSynth | |-----|----------------------|---------|----------| | 100 | Real | 12.67 | 14.46 | | 100 | Vanilla Syn. Aug. | 14.34 | 17.54 | | 100 | Synthio | 31.26 | 27.32 | | 100 | w/ Template Captions | 29.31 | 26.62 | | 100 | w/ ERM | 24.15 | 21.54 | | 200 | Real | 10.13 | 9.4 | | 200 | Vanilla Syn. Aug. | 12.55 | 12.91 | | 200 | Synthio | 21.87 | 16.16 | | 200 | w/ Template Captions | 20.31 | 15.82 | | 200 | w/ ERM | 17.14 | 13.04 | Table 9: Comparison of our trained Stable Diffusion model on AudioCaps test set | Model | FAD PANN ( ↓ ) | FAD VGG ( ↓ ) | IS PANN ( ↑ ) | CLAP LAION ( ↑ ) | |-------------------------------------------|------------------|-----------------|-----------------|--------------------| | AudioLDM2-large | 32.5 | 1.89 | 8.55 | 0.45 | | Tango-Full0FT-AC | 18.47 | 2.19 | 8.80 | 0.57 | | Tango 2 | 17.19 | 2.54 | 11.04 | 0.52 | | Make-an-Audio 2 | 11.75 | 1.8 | - | 0.6 | | Stable Audio VECaps (ours) | 15.12 | 2.21 | 15.07 | 0.57 | | Stable Audio VECaps + AudioCaps-FT (ours) | 14.93 | 2.19 | 15.42 | 0.56 | ## A.4.2 AUDIO GENERATION RESULTS FOR OUR TRAINED STABLE DIFFUSION Table 9 presents a comparison of audio generation results across several evaluation metrics. We evaluate our trained Stable Diffusion model (used in our experiments, including a version further fine-tuned on AudioCaps) against other available models and baselines from the literature. Notably, our model performs competitively with other fully open-source models across most metrics. ## A.4.3 FAD SCORES FOR GENERATED AUGMENTATIONS To offer an alternative perspective on the distributional consistency between the generated augmentations and the ground-truth small-scale dataset, we compare the Fr´ echet Audio Distance (FAD) scores (Kilgour et al., 2018). For this experiment, we use Synthio with Template Captions. Table 10 presents a comparison of FAD scores between Synthio and other baselines. Synthio achieves the highest FAD score, indicating that it produces the most consistent audio augmentations. Table 10: Comparison of FAD score of Vaniall Syn. Aug. and Stable Audio VECaps (ours). | n | Dataset | Model | FAD VGG ( ↓ ) | |-----|-----------|-----------------------------------------------------|-----------------| | 100 | NSynth | Vanilla Syn. Aug. Stable Stable Audio VECaps (ours) | 1.83 1.42 | | 200 | TUT | Vanilla Syn. Aug. Stable Audio VECaps (ours) | 1.71 1.45 | ## A.5 DATASET DETAILS NSynth Instruments: NSynth is a large-scale dataset consisting of musical notes played by a variety of instruments. It includes a rich set of acoustic features from instruments like guitars, flutes, and more, providing diverse sound textures for classification tasks. TUT Urban: The TUT Urban dataset captures everyday sounds from urban environments, including noises like traffic, human activities, and construction. It is commonly used for acoustic scene classification and environmental sound recognition. ESC-50 : ESC-50 is a well-known dataset for environmental sound classification, containing 50 categories of everyday sounds such as animal noises, natural elements, and human activities, making it suitable for multi-class classification challenges. UrbanSound8K (USD8K): USD8K is a curated collection of urban sounds divided into ten classes, including sirens, street music, and car horns. It is used widely for evaluating models on sound event detection in real-world scenarios. GTZAN: GTZAN is a music genre classification dataset that includes ten music genres such as pop, rock, and jazz. It is a standard benchmark for evaluating music classification models, although it has known data quality issues. Medley-solos-DB: This dataset consists of solo recordings of different musical instruments, making it valuable for studying isolated instrument sounds and training models for music instrument recognition. MUSDB18: MUSDB18 is used primarily for music source separation tasks. It contains full-track recordings of different music styles, providing a mix of vocals, drums, bass, and other instruments, useful for multi-class classification. DCASE Task 4: Part of the DCASE challenge, this dataset focuses on domestic sound scene and event classification. It includes various audio clips recorded in home environments, often used for anomaly detection and sound event classification. Vocal Sounds (VS): This dataset includes various vocal sounds such as singing, speech, and vocal effects, providing rich data for studying voice classification and enhancing models for vocal audio recognition tasks.