## Diagram: Token Routing Optimization in Batch Processing
### Overview
The diagram compares **original** and **optimized** expert load configurations in a token routing system. It illustrates how token-level and batch-level routing paths are restructured to improve computational efficiency. Two batches (Batch1, Batch2) are processed through grids representing token-expert assignments, with visual emphasis on activated (pink) vs. inactive (white) experts.
---
### Components/Axes
1. **Routing Paths**:
- **Token-level Routing Path**: Arrows connecting token grids to a "Token Scheduler" (central element).
- **Batch-level Routing Path**: Arrows connecting batch grids to a "New" routing output.
2. **Grids**:
- **Original Expert Loads**:
- Two grids (Batch1, Batch2) with 4x4 cells (tokens t1-t8, experts e1-e4).
- Pink cells indicate activated experts (all experts active per batch).
- **Optimized Expert Loads**:
- Two grids (New Batch1, New Batch2) with 4x4 cells.
- Pink cells indicate activated experts (half per batch).
3. **Labels**:
- **Original**: "Activate all experts per batch", "Each expert compute 1 token".
- **Optimized**: "Activate half experts per batch", "Each expert compute 2 tokens".
---
### Detailed Analysis
1. **Original Expert Loads**:
- **Batch1**: 4 experts (e1-e4) active, each handling 1 token (t1-t4).
- **Batch2**: 4 experts (e5-e8) active, each handling 1 token (t5-t8).
- **Token Scheduler**: Distributes tokens evenly across all experts.
2. **Optimized Expert Loads**:
- **New Batch1**: 2 experts (e1, e2) active, each handling 2 tokens (t1, t2 and t7, t3).
- **New Batch2**: 2 experts (e3, e4) active, each handling 2 tokens (t6, t4 and t8, t5).
- **Token Scheduler**: Consolidates token assignments to reduce active experts.
---
### Key Observations
1. **Expert Activation**:
- Original: 100% expert utilization (4 experts/batch).
- Optimized: 50% expert utilization (2 experts/batch).
2. **Token Computation**:
- Original: 1 token/expert.
- Optimized: 2 tokens/expert (doubled efficiency).
3. **Routing Changes**:
- Token-level routing remains unchanged (e1-e4).
- Batch-level routing shifts to "New" batches with consolidated expert assignments.
---
### Interpretation
The diagram demonstrates a **resource optimization strategy** for parallel processing:
- **Efficiency Gains**: By halving active experts and doubling tokens per expert, the system reduces overhead (e.g., memory, inter-expert communication) while maintaining throughput.
- **Scalability**: The "Token Scheduler" acts as a load balancer, ensuring tokens are distributed to maximize expert utilization without redundancy.
- **Trade-offs**: Fewer active experts may limit parallelism but improves per-expert throughput, ideal for memory-constrained environments.
This approach aligns with principles of **model parallelism optimization**, critical for deploying large language models (LLMs) on distributed hardware.
