## Diagram: Music SSL Representation and Classification Architecture
### Overview
This diagram illustrates a deep learning architecture designed for processing song waveforms. The system extracts features using pre-trained Self-Supervised Learning (SSL) models, processes these features through a downsampling module, and integrates them with category-specific metadata to perform classification tasks via a Transformer-based architecture.
### Components/Axes
The diagram utilizes a legend to define specific operations and model states:
* **Concatenate fusion (C)**: Indicated by a circle with a 'C'.
* **Additive fusion (+)**: Indicated by a circle with a '+' sign.
* **Frozen (Snowflake icon)**: Indicates model components that are not updated during training.
* **Training (Fire icon)**: Indicates model components that are updated during training.
### Detailed Analysis
#### 1. SSL Representations Extraction (Left Region)
* **Input**: "Song Waveform".
* **Processing**: The waveform is fed into two parallel, "Frozen" (non-trainable) SSL models: **MuQ** and **MusicFM**.
* **Temporal Resolution**: Each model outputs two distinct temporal resolutions:
* **30s**: Represented as 14 chunks.
* **420s**: Represented as a single block.
* **Fusion**: The outputs from both models (at both temporal resolutions) are combined at a **Concatenate fusion (C)** node.
#### 2. Downsampling Module (Center Region)
* **Input**: The concatenated output from the previous stage flows into the "Fused SSL Representation" block, which then feeds into the "Downsampling Module".
* **Architecture**: The module splits into two parallel paths:
* **Left Path**: "Depthwise Conv" followed by "Pointwise Conv".
* **Right Path**: "AvgPool1d" followed by "Pointwise Conv".
* **Fusion**: The outputs of these two paths are combined via **Additive fusion (+)**.
* **Output**: The result is labeled as "Downsampled Fused SSL Representation".
#### 3. Transformer and Classification Heads (Right Region)
* **Transformer Input**: The "Downsampled Fused SSL Representation" is fed into a **4-Layer Transformer**.
* **Metadata Integration**:
* "Data Source Category Label" is processed through a "Token Embedding & Expand In Time" module.
* This creates a "Data Source Category Representation".
* This representation is combined with the transformer input via **Additive fusion (+)**.
* **Outputs**: The 4-Layer Transformer feeds into two distinct "Training" heads:
* **Boundary Head**
* **Function Head**
### Key Observations
* **Transfer Learning**: The use of "Frozen" SSL models (MuQ and MusicFM) indicates a transfer learning approach where pre-trained audio features are utilized rather than trained from scratch.
* **Multi-Scale Processing**: The architecture explicitly handles different temporal scales (30s vs 420s), allowing the model to capture both short-term and long-term audio context.
* **Hybrid Input**: The system is not purely audio-based; it incorporates external metadata ("Data Source Category Label") to condition the Transformer's output.
* **Computational Efficiency**: The "Downsampling Module" suggests that the raw concatenated SSL representations are too high-dimensional for the Transformer, necessitating a reduction step before the main processing block.
### Interpretation
This architecture is designed for complex music information retrieval (MIR) tasks. By combining frozen, high-level audio representations (MuQ/MusicFM) with a trainable downsampling and transformer pipeline, the model effectively balances pre-existing audio knowledge with task-specific learning.
The inclusion of "Boundary Head" and "Function Head" suggests this model is likely performing two simultaneous tasks:
1. **Boundary Detection**: Identifying structural changes in the music (e.g., verse-chorus transitions).
2. **Function Classification**: Identifying the role or type of the audio segment (e.g., intro, outro, solo).
The "Data Source Category" input acts as a conditional bias, likely helping the model adjust its predictions based on the specific dataset or genre context from which the audio originated.
