## Diagram: Audio Dataset Production Workflow and Baseline Architecture
### Overview
The diagram illustrates two interconnected components:
1. **Dataset Production Workflow** (left section)
2. **Baseline Architecture** (right section)
Both sections emphasize feature extraction, transformation, and machine learning processes for audio data.
---
### Components/Axes
#### (a) Dataset Production Workflow
1. **Input**:
- Raw Audio (labeled as "1 Raw Audio")
- Labels (labeled as "3 Label")
2. **Processing Steps**:
- **Manual Annotation**: Human-labeled audio segments.
- **Drop Segmentation**: Rule-based segmentation by volume thresholds (e.g., "Drop 1", "Drop i", "Drop n").
- **Time-to-Frequency Transformations**:
- `mel-spectrogram`
- `cqt-chromagram`
- `vqt-chromagram`
- **Feature Stacking**: Combines transformed features into "Clip Feature".
3. **Output**:
- Labeled dataset with segmented audio clips and extracted features.
#### (b) Baseline Architecture
1. **Input**:
- Clip Features (from dataset production workflow).
2. **Feature Extraction Module**:
- **CNN or ViT**: Processes sub-clip features.
- **Transformer Encoder**:
- **Multi-Head Attention**: Captures contextual relationships.
- **Feed Forward**: Non-linear transformations.
- **Trainable Modules**: Learnable parameters for feature refinement.
3. **Output**:
- **Label Prediction**: Final classification output (`y_i`).
- **MLP Last-Layer Feature Visualization**: Visualizes intermediate features as clusters (e.g., green, blue, red dots).
---
### Detailed Analysis
#### Dataset Production Workflow
- **Drop Segmentation**: Segments raw audio into 7.5-second clips (e.g., "Clips 1", "Clips j").
- **Feature Extraction**:
- `mel-spectrogram`: Captures frequency magnitude over time.
- `cqt-chromagram`: Represents pitch-class content.
- `vqt-chromagram`: Visualizes harmonic structure.
- **Stacking**: Combines multiple features (e.g., mel, CQT, VQT) into a unified "Clip Feature".
#### Baseline Architecture
- **Transformer Encoder**:
- **Multi-Head Attention**: Processes sub-clip features in parallel, capturing long-range dependencies.
- **Feed Forward**: Applies non-linear transformations to attention outputs.
- **MLP**: Maps encoded features to label predictions (`y_i`).
- **Feature Visualization**:
- **Sub-Clip Features**: Represented as 2D grids (e.g., 128x128).
- **Clip Features**: Stacked 3D tensors with dimensions:
- `hop length*` (time resolution)
- `clip length` (frequency resolution)
- `number of mel-banks` (spectral bins).
---
### Key Observations
1. **Integration of Techniques**:
- Combines traditional audio features (mel-spectrogram, CQT) with deep learning (transformer, MLP).
2. **Hyper-Parameters**:
- `hop length*`: Controls time resolution in feature extraction.
- `clip length`: Defines frequency resolution.
3. **Rule-Based Segmentation**:
- Segments audio based on volume thresholds, ensuring consistent clip lengths.
4. **Visualization**:
- MLP last-layer features are clustered (e.g., green, blue, red), suggesting class separation.
---
### Interpretation
1. **Dataset Production**:
- The workflow emphasizes preprocessing raw audio into structured, labeled clips with multi-modal features. This ensures robustness for downstream tasks.
2. **Baseline Architecture**:
- The transformer encoder and MLP highlight the importance of contextual modeling and non-linear transformations for accurate label prediction.
3. **Feature Visualization**:
- The clustered visualization of MLP features suggests effective feature discrimination, critical for model interpretability.
4. **Hyper-Parameter Tuning**:
- Adjusting `hop length` and `clip length` balances temporal and spectral resolution, impacting model performance.
This diagram underscores a pipeline that bridges raw audio data with advanced machine learning, leveraging both traditional signal processing and deep learning techniques.
