## Diagram: LLaMA 70B + GH200 Memory Offload Architecture
### Overview
The diagram illustrates the memory offload strategy for the LLaMA 70B model using a GH200 accelerator. It shows the distribution of model weights between host memory (HBM) and GPU memory, with specific offload ratios (OR) for different weight components. The total offload ratio is 36%, with detailed breakdowns for three key weight types: `qkv_weights`, `kvcache`, and `outproj_weights`.
### Components/Axes
- **Left Section**:
- **LLaMA 70B**: Model name with ~150GB memory (red text).
- **GH200**: Accelerator with 96GB HBM (red text).
- **Offload Ratio (OR)**: Total OR = 36% (blue text).
- **Right Section**:
- **Legend**:
- Green squares: Weights on host memory.
- Blue squares: Weights on GPU memory.
- **Weight Components**:
1. **qkv_weights**:
- SplitK_GEMM (OR = 75%).
- 12 green (host) + 4 blue (GPU) blocks.
2. **kvcache**:
- SplitK_FlashAttn (OR = 25%).
- 3 green (host) + 9 blue (GPU) blocks.
3. **outproj_weights**:
- SplitK_GEMM (OR = 33%).
- 2 green (host) + 6 blue (GPU) blocks.
### Detailed Analysis
- **Memory Distribution**:
- **qkv_weights**: 75% host (green) vs. 25% GPU (blue).
- **kvcache**: 25% host (green) vs. 75% GPU (blue).
- **outproj_weights**: 33% host (green) vs. 67% GPU (blue).
- **Color Consistency**:
- Green blocks in all sections align with the "host memory" legend.
- Blue blocks in all sections align with the "GPU memory" legend.
- **Spatial Layout**:
- LLaMA 70B and GH200 are positioned on the far left.
- Arrows point to the three weight components on the right.
- Legend is placed at the top-right corner.
### Key Observations
1. **Highest Host Dependency**: `qkv_weights` rely most on host memory (75% OR).
2. **Lowest Host Dependency**: `kvcache` uses the least host memory (25% OR).
3. **Moderate Host Usage**: `outproj_weights` have a balanced split (33% host).
4. **Total Offload Ratio**: The overall OR of 36% suggests a hybrid approach, balancing host and GPU memory usage.
### Interpretation
The diagram demonstrates a memory optimization strategy for large language models. By offloading weights to host memory (HBM), the system reduces GPU memory pressure while maintaining performance. The use of **SplitK_GEMM** and **SplitK_FlashAttn** techniques tailors the OR for different weight types:
- **qkv_weights** (75% OR): Prioritize host memory for large query-key-value matrices.
- **kvcache** (25% OR): Maximize GPU memory for attention cache efficiency.
- **outproj_weights** (33% OR): Balance host/GPU usage for output projection layers.
The total OR of 36% reflects a system-wide trade-off, leveraging the GH200's 96GB HBM to complement LLaMA's 150GB requirements. This approach highlights the importance of component-specific memory management in large-scale AI inference.
