## Line Chart: Per-GPU Throughput vs. Batch Size
### Overview
The chart compares the performance of different GPU configurations (SGLang BF16, HF Static FP16, HF Dynamic FP16, HF KIVI INT4, and Kitty-Pro) in terms of per-GPU throughput (tokens processed per second) across varying batch sizes (0–250). The y-axis represents throughput in tokens/second, while the x-axis represents batch size. Data points are plotted with distinct colors for each configuration, and trends indicate how throughput scales with batch size.
---
### Components/Axes
- **Title**: "Per-GPU Throughput vs. Batch Size"
- **X-axis (Batch Size)**:
- Labels: 0, 50, 100, 150, 200, 250
- Scale: Linear increments of 50
- **Y-axis (Per-GPU Tokens/Second)**:
- Labels: 0, 500, 1000, 1500, 2000, 2500, 3000
- Scale: Linear increments of 500
- **Legend**:
- Position: Bottom-right corner
- Entries:
- **SGLang BF16**: Purple line with circular markers
- **HF Static FP16**: Blue line with circular markers
- **HF Dynamic FP16**: Orange line with circular markers
- **HF KIVI INT4**: Green line with circular markers
- **Kitty-Pro**: Red line with circular markers
---
### Detailed Analysis
1. **SGLang BF16 (Purple)**:
- Starts at ~600 tokens/sec at batch size 1.
- Rises sharply to ~1,700 tokens/sec at batch size 50.
- Peaks at ~3,000 tokens/sec at batch size 100.
- Declines slightly to ~2,900 tokens/sec at batch size 150.
- Stabilizes near ~3,000 tokens/sec at batch sizes 200 and 250.
2. **Kitty-Pro (Red)**:
- Begins at ~200 tokens/sec at batch size 1.
- Increases steadily to ~1,500 tokens/sec at batch size 50.
- Rises to ~1,700 tokens/sec at batch size 100.
- Continues upward to ~1,900 tokens/sec at batch size 150.
- Reaches ~2,100 tokens/sec at batch size 200.
- Ends at ~2,200 tokens/sec at batch size 250.
3. **HF Static FP16 (Blue)**:
- Starts at ~1,000 tokens/sec at batch size 1.
- Drops to ~500 tokens/sec at batch size 50.
- Further declines to ~300 tokens/sec at batch size 100.
- Stabilizes at ~250 tokens/sec at batch sizes 150–250.
4. **HF Dynamic FP16 (Orange)**:
- Begins at ~500 tokens/sec at batch size 1.
- Peaks at ~600 tokens/sec at batch size 50.
- Declines to ~400 tokens/sec at batch size 100.
- Further drops to ~350 tokens/sec at batch size 150.
- Ends at ~300 tokens/sec at batch size 250.
5. **HF KIVI INT4 (Green)**:
- Remains flat at ~200–300 tokens/sec across all batch sizes.
---
### Key Observations
- **SGLang BF16** dominates in throughput, especially at larger batch sizes (>100), maintaining near-peak performance.
- **Kitty-Pro** shows consistent linear growth but lags behind SGLang BF16.
- **HF Static FP16** underperforms significantly after batch size 1, suggesting inefficiency at scale.
- **HF Dynamic FP16** and **HF KIVI INT4** exhibit minimal throughput, with HF KIVI INT4 being the least efficient.
---
### Interpretation
The data highlights **SGLang BF16** as the most scalable and efficient configuration, achieving near-maximum throughput even at large batch sizes. **Kitty-Pro** demonstrates steady improvement but remains suboptimal compared to SGLang. The decline of **HF Static FP16** after small batch sizes suggests architectural limitations in handling larger workloads. **HF Dynamic FP16** and **HF KIVI INT4** underperform across all batch sizes, indicating potential inefficiencies in their design or implementation. These trends underscore the importance of configuration choice for optimizing GPU throughput in large-scale processing tasks.
