# EDMFormer: Genre-Specific Self-Supervised Learning for Music Structure Segmentation
**Authors**: Sahal Sajeer, Krish Patel, Oscar Chung, Joel Song Bae
## Abstract
Music structure segmentation is a key task in audio analysis, but existing models perform poorly on Electronic Dance Music (EDM). This problem exists because most approaches rely on lyrical or harmonic similarity, which works well for pop music but not for EDM. EDM structure is instead defined by changes in energy, rhythm, and timbre, with different sections such as buildup, drop, and breakdown. We introduce EDMFormer, a transformer model that combines self-supervised audio embeddings using an EDM-specific dataset and taxonomy. We release this dataset as EDM-98: a group of 98 professionally annotated EDM tracks. EDMFormer improves boundary detection and section labelling compared to existing models, particularly for drops and buildups. The results suggest that combining learned representations with genre-specific data and structural priors is effective for EDM and could be applied to other specialized music genres or broader audio domains.
## I Introduction
Music structure analysis (MSA) is a core task in music information retrieval (MIR), enabling applications like automatic song summarization, playlist generation, DJ assistance, and music recommendation [6]. Modern MSA systems often rely on self-supervised learning (SSL) to create audio embeddings from large collections of music [5]. Transformer-based models such as SongFormer perform well by learning harmonic and melodic similarity patterns across time [3].
However, these models assume that structure is mainly defined by harmonic repetition and lyrical phrasing, such as in Western pop music (e.g., verse, chorus, bridge). This assumption fails for EDM, where structure is driven by different features. EDM sections such as buildup and breakdown are not characterized by chords or vocals, but by spectral brightness, rhythmic density, and low-frequency energy changes [9]. As a result, even top-tier MSA models often mislabel EDM sections, revealing a systematic genre mismatch.
### I-A Motivation
Prior MIR research shows that genre mismatches reduce model performance when learned representations do not match the structural patterns of the target genre [4]. For EDM, switch point detection (SPD) studies indicate that energy novelty, drum onset counts, and timbral features are the best cues for structural transitions [8]. These cues are mostly absent in the current SSL embeddings trained on pop music. It is also noted that rhythmic and energy structuring can improve boundary detection by over 3% when combined with learned embeddings [8].
These observations suggest that SSL-based MSA models miss important information needed for energy-driven music due to an inadequate taxonomy and lack of relevant data. This finding motivates an EDM-specific approach that integrates a more appropriately structured taxonomy and data.
### I-B Related Works
Early MSA approaches relied on handcrafted features such as chroma and novelty curves to detect structural boundaries [6]. With the rise of deep learning, convolutional and recurrent neural networks replaced feature engineering with learned representations. More recently, however, transformer-based models train with SSL objectives, such as MuQ and MusicFM, enabling models like SongFormer to achieve high performance on pop music datasets [10, 7, 3].
However, improved representation learning does not necessarily translate to optimal structural segmentation. Several studies report that SSL-only embeddings underperform when domain-specialized rhythmic cues are ambiguous. Beat-focused feature research shows that explicit rhythmic taxonomy setups can significantly improve boundary detection performance, particularly when fused with neural embeddings [8]. Similarly, Joung et al. demonstrate that domain specialization improves model robustness in MIR tasks [4].
For EDM, SPD research identifies energy novelty, drum onset counts, and spectral flux as the strongest indicators of structural change [8]. But no existing MSA work has incorporated these findings into a domain-specific structure segmentation taxonomy. Additionally, existing datasets such as SALAMI still contain minimal EDM representation and use pop-centric taxonomies [2].
These gaps motivate the need for (1) an EDM dataset, (2) an EDM-specific structural taxonomy, and (3) a SSL transformer model that combines the two.
### I-C Problem Definition
Given an audio waveform $X$ , the goal of music structure segmentation is to predict a sequence of timestamps $T=\{t_1,t_2,...,t_n\}$ representing structural boundaries, along with corresponding section labels $Y=\{y_1,y_2,...,y_n\}$ .
Existing approaches typically learn a general mapping
$$
f_general:E_ssl(X)\xrightarrow{pop-centric dataset + general taxonomy}(T,Y), \tag{1}
$$
where $E_ssl(X)$ are embeddings extracted from pretrained self-supervised music models, trained on diverse music corpora.
In contrast, our approach defines a domain-specialized mapping
$$
f_edm:E_ssl(X)\xrightarrow{EDM-98 dataset + EDM taxonomy}(T,Y), \tag{2}
$$
where the same SSL embeddings are used, but training leverages the EDM-98 dataset and its EDM-specific structural taxonomy, allowing the model to learn accurate patterns and section labels.
## II Methodology
### II-A EDM-98 Dataset
We curated EDM-98, a dataset of 98 EDM tracks across five tempo categories (120-180 BPM) using its source corpus EDM-CUE, which contains 4710 songs [1]. Two professional annotators labelled structural sections (intro, build-up, drop, breakdown, outro, silence, and end) with ±0.5s precision. The dataset was constructed as a stratified, equal‑allocation subset to counter the heavy tempo imbalance of EDM‑CUE.
For our implementation, each track’s BPM was extracted (with tracks lacking BPM excluded) and then assigned to fixed buckets. The sample allocation was distributed as evenly as possible across these buckets, with residual capacity handling to reach N=98. Selection within each stratum was randomized without replacement.
The resulting EDM‑98 marginal tempo distribution was intentionally close to uniform across mid/high‑tempo strata (House/Deep House 19.4%; Electro/Dance, Techno, Trance/Progressive, Dubstep/Bass, Drum & Bass/Jungle each 14.3%; Chill/Downtempo 9.2%). This distribution produced a broad BPM spread (mean 137.31, median 135.00, SD 20.04, range 100-175).
In contrast, the full EDM‑CUE dataset is highly skewed toward high‑tempo Drum & Bass/Jungle, with the train split (N=4710) showing BPM with mean 169.52, median 174.0, and SD 11.62. 90.5% of all tracks are this subgenre [1]. This severe class imbalance yields a markedly different marginal tempo distribution than EDM-98, which intentionally flattens the tempo histogram to support robust model training and evaluation across a wider tempo/genre spectrum. Thus, EDM‑98 should be interpreted as a balanced, purpose‑built subset rather than a proportional microcosm of EDM‑CUE.
### II-B EDM-Specific Taxonomy
All tracks in EDM-98 were labelled using an EDM-specific taxonomy designed to reflect changes in energy. As traditional pop-oriented taxonomies (verse, chorus, bridge) failed to capture EDM’s structure, we defined a new EDM taxonomy with the following categories:
- Intro: Beginning of the track, low energy, sparse instrumentation.
- Build-up: Gradually increasing energy and tension, often with rising drum patterns or risers.
- Drop: Peak energy section, featuring main rhythm motifs and basslines.
- Breakdown: Reduced energy sections, often melodic or atmospheric, providing contrast to drops.
- Outro: Ending portion, with gradually decreasing energy.
- Silence: Sections with minimal or no audio signal.
- End: Explicit track ending or final note.
Figure 1 illustrates an example track annotated with our proposed taxonomy, highlighting EDM’s structural transitions.
<details>
<summary>2603.08759v1/annotations_example.png Details</summary>
### Visual Description
## Line Chart: Waveform Analysis of "Afterglow" by Wilkinson Becky Hill
### Overview
The chart displays a time-domain waveform analysis of the audio track "Afterglow" by Wilkinson Becky Hill. It visualizes amplitude variations over time, segmented into labeled sections with distinct colors. The waveform exhibits dynamic changes in intensity, reflecting the song's structural progression.
### Components/Axes
- **X-axis**: Time (Seconds), ranging from 0 to 225 seconds, divided into 10-second intervals.
- **Y-axis**: Amplitude (normalized to ±1.0), with no explicit scale but visible peaks/troughs.
- **Legend**: Located at the bottom, mapping colors to sections:
- Blue: Intro
- Yellow: Buildup
- Red: Drop
- Purple: Breakdown
- Green: Outro
- Gray: END
- **Labels**: Section names (e.g., "Intro," "Buildup") are positioned above their respective colored segments.
### Detailed Analysis
1. **Intro (0–30s, Blue)**:
- Amplitude remains low (~0.1–0.3), with minor fluctuations.
- Sudden peak at ~25s (~0.5) before returning to baseline.
2. **Buildup (30–40s, Yellow)**:
- Gradual amplitude increase from ~0.2 to ~0.6.
- Sustained oscillations with moderate intensity.
3. **Drop (40–100s, Red)**:
- Dominant high-amplitude oscillations (~0.7–0.9).
- Irregular peaks and troughs, indicating rhythmic complexity.
4. **Breakdown (100–130s, Purple)**:
- Amplitude drops to ~0.3–0.5, with reduced frequency of peaks.
- Sparse, isolated spikes suggest a quieter, fragmented section.
5. **Buildup (130–150s, Yellow)**:
- Rapid amplitude rise from ~0.4 to ~0.8.
- Sharp, repetitive peaks mirroring the initial Buildup.
6. **Drop (150–200s, Red)**:
- Similar to the first Drop, with sustained high-amplitude oscillations (~0.7–0.9).
- Increased density of peaks compared to the first Drop.
7. **Outro (200–225s, Green)**:
- Gradual amplitude decay from ~0.6 to ~0.1.
- Final sharp drop to zero at 225s, marked by the "END" label.
### Key Observations
- **Structural Symmetry**: The first and second Drops (40–100s and 150–200s) exhibit comparable amplitude ranges but differ in peak density.
- **Dynamic Contrast**: The Breakdown (100–130s) serves as a transitional "quiet" section between high-energy Drops.
- **Amplitude Extremes**: The maximum amplitude (~0.9) occurs in both Drops, while the minimum (~0.1) is in the Intro and END.
- **Temporal Precision**: The END section’s abrupt termination at 225s suggests a programmed cutoff rather than a natural fade.
### Interpretation
The waveform reveals a deliberate compositional structure:
- **Intro** establishes a baseline, building tension through the **Buildup**.
- **Drop** sections (40–100s and 150–200s) represent climactic moments, with the second Drop showing increased rhythmic complexity.
- The **Breakdown** acts as a narrative pause, contrasting with the preceding intensity.
- The **Outro**’s gradual fade and abrupt END suggest a technical conclusion, possibly indicating a loop or abrupt silence in the original audio.
The chart underscores the interplay between amplitude and time, mapping the song’s emotional arc through technical parameters. The use of color-coded segments aligns with common DAW (Digital Audio Workstation) workflows, where sections are visually distinguished for editing or analysis.
</details>
Figure 1: Example track segment annotated with proposed EDM taxonomy.
### II-C Model Architecture
Our model architecture largely followed SongFormer’s design, with the key differences being our incorporation of an EDM-specific taxonomy, a training stage using EDM-98, and a final fine-tuning stage on EDM-98 [3].
We extracted self-supervised music representations from two pretrained audio foundation models, MuQ and MusicFM [10, 7]. We selected these two as our foundation models because they both produce temporally rich embeddings from relatively large audio corpora, providing strong low-level acoustic representations. Critically, however, neither model’s pretraining is optimized for EDM structure. Their learned representations reflect general acoustic patterns rather than the energy and timbral transitions that define EDM sections. This generality is precisely why domain-specific fine-tuning over our dataset EDM-98 is necessary. EDMFormer repurposed these general embeddings using an EDM-specific framework, teaching the model which acoustic patterns are structurally meaningful for the genre.
We fused both models as they differ in their pretraining objectives: MuQ uses masked prediction over Mel-quantized spectral targets, emphasizing timbral and spectral features, while MusicFM uses masked token modeling with random projection and has demonstrated stronger performance on longer-range structural tasks due to its 30-second input context during pretraining [10, 7]. Our combination of the two methods provides complementary information to the downstream transformer. Each model produced 1024-dimensional embeddings computed at two temporal contexts (30 second and 420 second windows), resulting in four embedding streams per track. The four embeddings were then concatenated to form a 4096-dimensional representation. Next, this concatenated 4096-dimensional representation was projected to 2048 dimensions through a linear layer and provided as the input to the SongFormer encoder.
Figure 2 visualizes SongFormer’s model architecture in greater detail for reference.
<details>
<summary>2603.08759v1/model_architecture.png Details</summary>
### Visual Description
## Flowchart: Audio Processing and Classification Pipeline
### Overview
The diagram illustrates a multi-stage audio processing pipeline for music classification, combining waveform analysis, SSL (Self-Supervised Learning) representations, and transformer-based modeling. Key components include MuQ and MusicFEM modules, SSL representation fusion, and a 4-layer transformer with specialized heads.
### Components/Axes
1. **Input Stage**:
- **Song Waveform**: Raw audio input (time-domain signal).
- **MuQ Block**: Processes 30s segments (14 channels) with additive fusion.
- **MusicFEM Block**: Processes 420s segments with frozen parameters.
2. **SSL Representation Extraction**:
- **Concatenate Fusion**: Combines MuQ and MusicFEM outputs (30s + 420s).
- **Downsampling Module**: Reduces dimensionality via depthwise and pointwise convolutions.
3. **Transformer Processing**:
- **Fused SSL Representation**: Input to 4-layer transformer.
- **Boundary Head**: Predicts temporal boundaries (orange flame icon).
- **Function Head**: Extracts functional features (orange flame icon).
4. **Output Stage**:
- **Token Embedding & Expand In Time**: Converts fused SSL to temporal tokens.
- **Data Source Category Label**: Final classification output.
5. **Legend**:
- **Orange Flame**: Training phase.
- **Blue Snowflake**: Frozen parameters.
- **Red Cross**: Additive fusion.
### Detailed Analysis
- **MuQ/MusicFEM Blocks**:
- MuQ handles short-term features (30s, 14 channels), while MusicFEM captures long-term structure (420s).
- Both use additive fusion (⊕ symbol) for feature combination.
- **SSL Fusion**:
- Concatenation (⊕) merges MuQ and MusicFEM outputs, followed by downsampling via depthwise (↓) and pointwise (↓) convolutions.
- **Transformer Architecture**:
- 4-layer transformer processes fused SSL representations.
- Boundary and function heads specialize in temporal segmentation and feature extraction.
- **Tokenization**:
- "Expand In Time" block converts SSL representations into temporal tokens for transformer input.
### Key Observations
1. **Temporal Hierarchy**:
- Short-term (30s) and long-term (420s) features are processed separately before fusion.
2. **Parameter Efficiency**:
- MusicFEM and downsampling modules use frozen parameters (❄️), reducing computational cost.
3. **Multi-Task Heads**:
- Boundary and function heads enable joint prediction of segmentation and classification.
### Interpretation
This pipeline demonstrates a hybrid approach to music understanding:
1. **Feature Integration**: Combines MuQ's spectral analysis with MusicFEM's temporal modeling.
2. **Efficient Training**: Freezing MusicFEM parameters allows focused training on MuQ features.
3. **Transformer Adaptation**: Adapts SSL representations for music-specific tasks via specialized heads.
4. **Temporal Reasoning**: Token expansion enables the model to handle variable-length audio sequences.
The architecture suggests applications in music genre classification, mood detection, or structural analysis, with a focus on balancing computational efficiency and representational power.
</details>
Figure 2: SongFormer’s model architecture.
### II-D Baseline
We evaluated EDMFormer against the newest iteration of SongFormer (2025) as our primary baseline [3]. SongFormer is the current best SSL-based MSA model, having demonstrated strong performance on pop-centric benchmarks such as SALAMI [2]. It is pretrained on diverse music corpora using heterogeneous supervision and uses a general structural taxonomy (e.g. intro, verse, chorus, bridge, outro).
SongFormer serves as an ideal baseline for two reasons. First, it shares the same foundation as EDMFormer, allowing us to isolate the improvements that come with domain-specific data and taxonomy. Second, SongFormer reflects the current standard against which any MSA model should be measured. We evaluated SongFormer in a zero-shot transfer setting, applying it directly to our tracks from EDM-98 without fine-tuning, testing how well general MSA models can analyze EDM.
### II-E Evaluation
We evaluated results using hit rate (HR), which measures the percentage of predicted boundaries that fall within a tolerance window of a ground-truth boundary. The metrics in specific were HR@0.5s, HR@3s, and per-frame label accuracy (ACC) with an 88-10 train-test split across the EDM dataset. A 5-fold cross-validation was performed on the training set for hyperparameter tuning.
## III Results
Table I compares the performance of our proposed model EDMFormer against the industry’s baseline SongFormer (rounded to 3 decimal points).
TABLE I: EDMFormer vs. SongFormer Performance
| SongFormer (Pop Taxonomy) EDMFormer (EDM Taxonomy) | 0.569 0.616 | 0.608 0.635 | 0.148 0.883 |
| --- | --- | --- | --- |
As shown in Table I, EDMFormer outperformed SongFormer across all three evaluation metrics, with improvements of 4.7%, 2.7%, and 73.5% in HR@0.5, HR@3, and ACC, respectively. The most striking gain was in per-frame accuracy (ACC), where EDMFormer achieved 0.883 compared to SongFormer’s 0.148, a 73.5% improvement that reflects the fundamental mismatch between a pop-centric taxonomy and EDM’s structural patterns. Boundary detection improvements further confirm that EDMFormer learns more temporally precise transition cues, such as drop onsets and buildup peaks, which the pop-centric SongFormer systematically misses.
The magnitude of improvement across all three metrics supports the core hypothesis that genre mismatch is a primary driver of poor MSA performance on EDM. EDMFormer’s architecture, which fuses MuQ and MusicFM embeddings at two temporal scales before passing the combined representation to the SongFormer encoder, preserved the expressive power of general SSL embeddings while allowing the transformer to specialize its attention patterns toward energy-driven structural transitions. The consistent directional improvement across all metrics and folds indicates that the proposed combination of domain-specific data, taxonomy, and SSL-based architecture is an effective and reproducible approach for adapting music structure analysis to specialized genres.
## IV Conclusion
This research introduces the EDM-98 dataset, proposes an EDM-specific structural taxonomy, and demonstrates that self-supervised music representations with a transformer model can improve boundary detection and section labelling accuracy for EDM. These results indicate that combining genre-specific datasets and taxonomies with SSL-based transformer models is an effective approach for adapting MSA to specialized genres.
Several limitations constrain the current study. EDM-98 has only 98 tracks. While this amount is sufficient as a proof-of-concept, it represents only a small subset of the broader EDM-CUE corpus, with dataset growth bottlenecked by the time needed for high-precision manual annotation. Another limitation lies in the foundation models themselves: although MuQ and MusicFM provide strong general acoustic representations, their pretraining objectives are not optimized for the energy-driven structural logic of EDM, and may inadequately capture the timbral and dynamic transitions. Lastly, each track in EDM-98 was labeled by a single annotator, potentially introducing bias.
Future work can address these gaps through dataset scaling, integration of explicit acoustic cues, and inter-annotator agreement metrics. Future work might also include hybrid feature learning, hierarchical annotations, relative positional embeddings for long tracks, and real-time inference capabilities for DJ-oriented applications. Implementing these ideas would strengthen both the reliability and representational capacity of future iterations, with longer-term potential for domain-specific SSL pretraining and real-time inference for DJ-oriented applications.
## V Acknowledgements
We thank the annotators for their contributions to the development of the EDM-98 dataset and the open-source communities behind SongFormer, MuQ, MusicFM, and librosa. This research was supported through the mentorship and computational resources provided by the University of Waterloo’s AI team, WAT.ai. We also thank the Canadian Undergraduate Conference on Artificial Intelligence (CUCAI) for the opportunity to present this work.
## References
- [1] G. Argüello, L. Lanzendörfer, and R. Wattenhofer (2024) Cue point estimation using object detection. arXiv preprint arXiv:2407.06823v1. Cited by: § II-A, § II-A.
- [2] D. De Roure, J. S. Downie, and I. Fujinaga (2011) Structural analysis of large amounts of music information (salami). Proceedings of the International Society for Music Information Retrieval Conference. Cited by: § I-B, § II-D.
- [3] C. Hao et al. (2025) SongFormer: scaling music structure analysis with heterogeneous supervision. Cited by: § I-B, §I, § II-C, § II-D.
- [4] H. Joung and K. Lee (2024) Music auto-tagging with robust music representation learned via domain adversarial training. ICASSP. Cited by: § I-A, § I-B.
- [5] S. Liu et al. (2022) Audio self-supervised learning: a survey. Cited by: §I.
- [6] J. Paulus, A. Klapuri, et al. (2010) A state of the art report: audio-based music structure analysis. ISMIR. Cited by: § I-B, §I.
- [7] M. Won, Y. Hung, and D. Le (2023) A foundation model for music informatics. arXiv preprint arXiv:2311.03318. Cited by: § I-B, § II-C, § II-C.
- [8] M. Zehren et al. (2024) Interpretability of methods for switch point detection in electronic dance music. Cited by: § I-A, § I-B, § I-B.
- [9] H. T. Zeiner-Henriksen (2010) The ”poumtchak” pattern: correspondences between rhythm, sound, and movement in electronic dance music. University of Oslo. Cited by: §I.
- [10] H. Zhu et al. (2025) MuQ: self-supervised music representation learning with mel residual vector quantization. IEEE Transactions on Audio, Speech, and Language Processing. Cited by: § I-B, § II-C, § II-C.