## Diagram: Baseline vs. Proposed Architecture (Device-Level Component Breakdown)
### Overview
The image compares two computational architectures ("Baseline" and "Ours") across five devices (Device1–Device5). Each device is represented by horizontal bars segmented into colored blocks, with a legend mapping colors to components (e.g., Embedding, Dense, MoE) and stages (VPP stages 0–3). The Baseline uses simpler components, while "Ours" introduces specialized layers (MTP, MTP-N, MTP-K) and staged processing.
---
### Components/Axes
- **Y-Axis**: Devices (Device1–Device5), ordered top to bottom.
- **X-Axis**: Component stages (no explicit scale; relative lengths imply progression).
- **Legend**:
- **Colors**:
- Gray: Embedding (E)
- Purple: Dense (D), MoE (M)
- Blue: MTP (T)
- Brown: MTP-N (N), MTP-K (K), Loss (L)
- Dashed lines: Micro batch boundaries (#0, #1)
- **Stages**: VPP stages 0–3 (brown blocks).
---
### Detailed Analysis
#### Baseline Architecture
- **Device1**:
- `E D M M` (Embedding, Dense, two MoE layers).
- **Device2**:
- `D D M M` (Dense layers dominate; no Embedding).
- **Device3**:
- `M M M M` (Four MoE layers).
- **Device4**:
- `M M M M` (Same as Device3).
- **Device5**:
- `M M M M T L` (Four MoE layers, one MTP, one Loss).
#### Proposed Architecture ("Ours")
- **Device1**:
- `E D M M M M` (Embedding, Dense, four MoE layers).
- Brown blocks (VPP stages 0–3) appear in the final two MoE layers.
- **Device2**:
- `D D D D M M M M` (Four Dense layers, three MoE layers).
- Brown blocks in the final MoE layer.
- **Device3**:
- `M M M M M M` (Six MoE layers).
- Brown blocks in the final two MoE layers.
- **Device4**:
- `M M M M M M K K` (Six MoE layers, two MTP-K layers).
- Brown blocks in MoE and MTP-K layers.
- **Device5**:
- `M M M M M M L L` (Six MoE layers, two Loss layers).
- Brown blocks in MoE and Loss layers.
---
### Key Observations
1. **Component Distribution**:
- Baseline uses fewer specialized layers (no MTP variants).
- "Ours" introduces MTP-N, MTP-K, and Loss layers, suggesting task-specific optimizations.
2. **Stage Progression**:
- Brown blocks (VPP stages) cluster in later layers, implying staged processing (e.g., parallelization or optimization steps).
3. **Device-Specific Patterns**:
- Device5 in "Ours" has the most stages (MTP-K and Loss), indicating advanced processing.
- Baseline Device5 adds MTP and Loss, while "Ours" replaces these with staged MoE layers.
---
### Interpretation
The diagram illustrates a shift from a monolithic Baseline architecture to a modular, staged approach in "Ours." The introduction of MTP variants (T, N, K) and Loss layers suggests specialized processing for tasks like attention, normalization, or error correction. The VPP stages (brown blocks) likely represent parallel or optimized computation phases, with later stages handling complex operations (e.g., Loss computation). Device5 in "Ours" demonstrates the highest complexity, possibly reflecting a focus on end-to-end optimization. The absence of numerical values limits quantitative analysis, but the structural differences imply improved scalability or efficiency in the proposed method.
