## Line Chart: Per-GPU Throughput vs. Batch Size
### Overview
The chart compares the performance of six methods in terms of per-GPU throughput (tokens/second) across varying batch sizes (0–250). Throughput is plotted on the y-axis, and batch size on the x-axis. The legend identifies methods by color, with SGLANG INT4 BDQ (brown) showing the highest throughput, followed by SGLANG BF16 (purple) and Kitty-Pro (pink). HF methods (Static FP16, Dynamic FP16, KIVI INT4) exhibit lower performance, with some declining at larger batch sizes.
### Components/Axes
- **Title**: "Per-GPU Throughput vs. Batch Size"
- **Y-Axis**: "Per-GPU Tokens / Second" (0–3500, increments of 500)
- **X-Axis**: "Batch Size" (0–250, increments of 50)
- **Legend**: Located in the bottom-right corner, mapping colors to methods:
- Blue: HF Static FP16
- Orange: HF Dynamic FP16
- Green: HF KIVI INT4
- Pink: Kitty-Pro
- Purple: SGLANG BF16
- Brown: SGLANG INT4 BDQ
### Detailed Analysis
1. **SGLANG INT4 BDQ (Brown)**:
- Starts at ~600 tokens/sec (batch size 0).
- Increases steadily, reaching ~3800 tokens/sec at batch size 250.
- Maintains the highest throughput across all batch sizes.
2. **SGLANG BF16 (Purple)**:
- Begins at ~600 tokens/sec (batch size 0).
- Rises sharply to ~3000 tokens/sec by batch size 100.
- Plateaus slightly above 3000 tokens/sec for larger batches.
3. **Kitty-Pro (Pink)**:
- Starts at ~200 tokens/sec (batch size 0).
- Increases linearly to ~2200 tokens/sec at batch size 250.
4. **HF Static FP16 (Blue)**:
- Begins at ~200 tokens/sec (batch size 0).
- Peaks at ~1000 tokens/sec (batch size 50).
- Declines to ~800 tokens/sec at batch size 250.
5. **HF Dynamic FP16 (Orange)**:
- Starts at ~200 tokens/sec (batch size 0).
- Peaks at ~500 tokens/sec (batch size 50).
- Drops to ~300 tokens/sec at batch size 250.
6. **HF KIVI INT4 (Green)**:
- Starts at ~100 tokens/sec (batch size 0).
- Rises to ~250 tokens/sec (batch size 50).
- Declines to ~200 tokens/sec at batch size 250.
### Key Observations
- **SGLANG INT4 BDQ** dominates performance, especially at larger batch sizes (>100).
- **HF methods** (Static, Dynamic, KIVI) show diminishing returns or inefficiency at higher batch sizes.
- **Kitty-Pro** and **SGLANG BF16** maintain consistent growth but lag behind SGLANG INT4 BDQ.
- **HF Static FP16** and **HF Dynamic FP16** exhibit sharp declines after batch size 50, suggesting suboptimal scaling.
### Interpretation
The data highlights **SGLANG INT4 BDQ** as the most efficient method for high-throughput applications, particularly at larger batch sizes. The HF methods’ performance degradation at higher batches may stem from hardware constraints or algorithmic limitations. Kitty-Pro and SGLANG BF16 offer viable alternatives but with lower scalability. The chart underscores the importance of method selection based on batch size requirements, with SGLANG INT4 BDQ being optimal for maximizing throughput.
